S3FS mounting parameter optimization method, storage system, equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses an S3FS mounting parameter optimization method, a storage system, equipment and a medium, S3FS mounting parameters are optimized through a DQN algorithm, the adjustment direction of the mounting parameters is clearer, the parameter optimization efficiency is higher, and compared with a traditional optimization method depending on expert experience and manual debugging, the optimization efficiency is higher, and the optimization efficiency is higher. The operation is simpler.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an S3FS mount parameter optimization method, storage system, device and medium. Background Art

[0002] With the rapid development of cloud computing technology, object storage is increasingly widely used. The basic unit of object storage is an object, and each object contains data and metadata. Object storage systems usually have the characteristics of high availability, scalability, and easy management. In the cloud platform, object storage provides users with object storage services based on the S3 protocol. When using object storage, users sometimes need to mount the S3 bucket of object storage to the local file system and keep the original format of the object. When using S3FS to mount the file system, you need to execute the mount command. In addition to the basic mandatory parameters, there are also some non-mandatory parameters in the mount command. These non-mandatory parameters affect various aspects of the file system. If the mount parameters are used improperly, it will affect the use of the file system. For example, if the user uses the default mount parameters, the performance of the user client may not be fully utilized, or if the user uses incorrect mount parameters, the performance of the cloud object storage may not be fully utilized. Summary of the invention

[0003] In view of this, the present invention provides an S3FS mount parameter optimization method, storage system, device and medium to solve the problem of improper use of mount parameters that affects the use of the file system.

[0004] In a first aspect, the present invention provides an S3FS mount parameter optimization method for an S3FS mount parameter optimization system; the method comprises:

[0005] Obtain a user instruction, a mount key information set and a first S3FS mount parameter set; and obtain a target S3FS mount parameter set through DQN algorithm processing based on the user instruction, the mount key information set and the first S3FS mount parameter set.

[0006] The S3FS mount parameter optimization method provided by the present invention optimizes the S3FS mount parameters through the DQN algorithm, and the adjustment direction of the mount parameters is clearer, the parameter optimization efficiency is higher, and it is simpler than the traditional optimization method relying on expert experience and manual debugging.

[0007] In an optional implementation, the S3FS mount parameter optimization system includes an S3FS file system; based on a user instruction, a mount key information set and a first S3FS mount parameter set, a target S3FS mount parameter set is obtained through DQN algorithm processing, including:

[0008] Based on the mount key information set, the S3FS file system is mounted using the first S3FS mount parameter set; based on user instructions, the S3FS file system operation is controlled and it is determined whether the performance of the S3FS file system is improved; when the performance of the S3FS file system is improved, the DQN algorithm is used to optimize the first S3FS mount parameter set until the performance of the S3FS file system is not improved within a preset number of iterations, thereby obtaining a target S3FS mount parameter set.

[0009] The S3FS mount parameter optimization method provided by the present invention can mount and run the S3FS file system based on user instructions and key mount information. By continuously judging whether the performance is improved and continuing to optimize the parameters when the performance is improved, the performance of the S3FS file system can be significantly improved. At the same time, by using the DQN algorithm for parameter optimization, the need for manual intervention is reduced, and the problem that traditional file system parameter configuration often requires a lot of manual experience and repeated experiments is solved.

[0010] In an optional implementation, based on a user instruction, controlling the operation of the S3FS file system and determining whether the performance of the S3FS file system is improved includes:

[0011] Based on user instructions, the operation of the S3FS file system is controlled to obtain an operation data set; multiple business performance indicator values ​​are determined based on the operation data set; and whether the performance of the S3FS file system is improved is determined based on the multiple business performance indicator values.

[0012] The S3FS mount parameter optimization method provided by the present invention can comprehensively and accurately reflect the performance of the file system in actual business operations through the running data set, and then can more comprehensively and accurately reflect the performance of the file system from different angles through multiple business performance indicator values, and then can comprehensively judge the performance changes of the S3FS file system.

[0013] In an optional implementation, when the performance of the file system is improved, the first S3FS mount parameter set is optimized using the DQN algorithm until the performance of the S3FS file system is not improved within a preset number of iterations, and a target S3FS mount parameter set is obtained, including:

[0014] When the performance of the S3FS file system is improved, the DQN algorithm is used to optimize the first S3FS mount parameter set and control the S3FS file system to stop running, and a second S3FS mount parameter set is obtained; the S3FS file system is mounted using the second S3FS mount parameter set, and the step of controlling the operation of the S3FS file system is returned to, and the iteration is repeated until the performance of the S3FS file system is not improved within a preset number of iterations, and a target S3FS mount parameter set is obtained.

[0015] The S3FS mount parameter optimization method provided by the present invention optimizes the S3FS mount parameters by continuously monitoring the performance of the S3FS file system, so that the S3FS file system can dynamically adjust the parameters adaptively to achieve better performance. At the same time, a preset number of iterations is set to limit the optimization process, thereby avoiding the problem of over-optimization. Therefore, by implementing the present invention, the DQN algorithm is used to achieve parameter optimization, and combined with the operation of automatically controlling the operation and stop of the file system, the entire optimization process is highly automated and intelligent.

[0016] In an optional embodiment, the method further includes:

[0017] When the performance of the S3FS file system is not improved, the step of controlling the operation of the S3FS file system is returned, and the process is repeated until the performance of the S3FS file system is not improved within a preset number of iterations, thereby obtaining a target S3FS mount parameter set.

[0018] In a second aspect, the present invention provides a storage system, the system comprising: an S3FS client and an object storage service client;

[0019] The object storage service end is used to send user instructions, a mount key information set and a first S3FS mount parameter set to the S3FS client; the S3FS client includes an S3FS mount parameter optimization system, which is used to execute the S3FS mount parameter optimization method of the first aspect or any corresponding implementation method thereof.

[0020] The storage system provided by the present invention sends user instructions, a mount key information set, and a first S3FS mount parameter set to the S3FS client through the object storage service end. Further, based on these data, by executing the mount parameter optimization method on the S3FS client, the interaction between the S3FS client and the object storage service end can be better adapted. Therefore, by implementing the present invention, the user only needs to input the user instructions, the mount key information set, and the first S3FS mount parameter set into the storage system to complete the automatic optimization of the S3FS mount parameters, which is simpler than the traditional optimization method relying on expert experience and manual debugging.

[0021] In an optional implementation, the S3FS mount parameter optimization system includes: a business operation module, a business stop module, a mount parameter optimization module and an S3FS file system;

[0022] The business operation module is used to run the S3FS file system; the business stop module is used to stop the S3FS file system; the mount parameter optimization module is used to optimize the S3FS mount parameters of the S3FS file system to obtain the target S3FS mount parameter set.

[0023] The storage system provided by the present invention can optimize the S3FS mount parameters of the S3FS file system by combining a business operation module, a business stop module and a mount parameter optimization module.

[0024] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the S3FS mount parameter optimization method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0025] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the S3FS mount parameter optimization method of the first aspect or any corresponding embodiment thereof.

[0026] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the S3FS mount parameter optimization method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0028] Figure 1 is a flow chart of a conventional S3FS mount parameter optimization method according to an embodiment of the present invention;

[0029] Figure 2 is a structural block diagram of a storage system according to an embodiment of the present invention;

[0030] Figure 3 is a flow chart of a method for optimizing S3FS mount parameters according to an embodiment of the present invention;

[0031] Figure 4 is a flow chart of another S3FS mount parameter optimization method according to an embodiment of the present invention;

[0032] Figure 5 is a schematic diagram of a DQN algorithm application process according to an embodiment of the present invention;

[0033] Figure 6 is a flow chart of an S3FS mount parameter optimization method based on a DQN algorithm according to an embodiment of the present invention;

[0034] Figure 7 is a schematic diagram of a user interaction interface according to an embodiment of the present invention;

[0035] Figure 8 is a schematic diagram of a DQN algorithm execution flow according to an embodiment of the present invention;

[0036] Fig. 9 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0038] When using S3FS to mount a file system, you need to execute the mount command. In addition to the basic mandatory parameters, there are also some non-mandatory parameters in the mount command. These non-mandatory parameters affect various aspects of the file system. If the mount parameters are used improperly, the use of the file system will be affected.

[0039] First, if users use the default mount parameters, the performance of the user's client may not be fully utilized. For example, the "use_cache" parameter is disabled by default in the S3FS mount command. However, if this option is enabled, the read and write performance of the file will be better. When the user does not know that the mount parameters can be selected, the system will disable them by default. At this time, S3FS does not enable local caching. Although the default method can ensure that all data is stored in the cloud, in some scenarios where data security is not important, this mount parameter will affect performance.

[0040] Second, if users use incorrect mount parameters, the performance of cloud object storage may not be fully utilized. For example, parallel_count represents the number of parallel requests for uploading large objects. S3FS uploads large objects through segmented post requests and sends parallel requests. This option limits the number of parallel requests that S3FS requests at a time. It can set the maximum number of requests sent to the object storage service in parallel for each file read and write operation. The value needs to be set according to the CPU and network frequency band. By default, this value is set to 5. For very large objects, you can increase this value to get more throughput.

[0041] Furthermore, when using S3FS to mount object storage in different scenarios, there will be different usage methods, different mounting parameters, and different capabilities. If you want to maximize the capabilities of the object storage client and server, you need storage experts to develop different solutions for different scenarios and optimize them. Traditional S3FS mount parameter optimization methods are as follows: Figure 1 shown.

[0042] However, when using object storage, users have different requirements for system performance due to their different needs and businesses. For example, the read and write operations of large files have higher requirements for bandwidth, while the read and write operations of small files place more emphasis on IOPS performance. In order to better meet the needs of users in specific scenarios, it is necessary to communicate in depth with users, clarify the key indicators they are most concerned about, and write simulated business test programs based on the scale of business. This means that if users lack knowledge of networks, object storage, file systems, operating systems, etc., they may not be able to fully utilize the performance of object storage. Even if users have certain usage experience and knowledge reserves, they still need to invest a lot of time and energy to optimize the S3FS mount parameters.

[0043] According to an embodiment of the present invention, an embodiment of a method for optimizing S3FS mount parameters is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] In this embodiment, a method for optimizing S3FS mount parameters is provided, which can be used as follows: Figure 2 The S3FS mount parameter optimization system 111 in the S3FS client 11 is shown. Figure 3 is a flow chart of a method for optimizing S3FS mount parameters according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0045] Step S301, obtaining a user instruction, a mount key information set and a first S3FS mount parameter set.

[0046] S3FS is a file system interface that supports exporting buckets in object storage as files and is compatible with POSIX syntax. S3FS is a file system developed based on FUSE, which allows S3 buckets to be mounted to the local file system and keeps the original format of the object.

[0047] Specifically, the user instruction may include a service start command, a service stop command, and the like.

[0048] Furthermore, the mount key information set represents the core information set required to ensure that the S3FS file system can be successfully mounted to the object storage service client, which may include storage service client access information, mount path information, and other mount configuration information.

[0049] The storage service client access information may include:

[0050] (1) Storage service endpoint (URL): For example, when using Amazon S3 service, it can be an endpoint address similar to "https: / / s3.amazonaws.com" used to locate the storage service.

[0051] (2) Bucket name: represents the basic storage unit name in object storage, such as "my-company-data-bucket". Files in the file system are actually stored in these buckets.

[0052] (3) Access Key and Secret Key or other authentication information: used for identity authentication to ensure that only authorized users can access and operate data in the bucket. In some environments, other authentication methods may be used, such as role-based access control (RBAC)-related tokens.

[0053] The mount path information may be a local mount path, which indicates a mount point specified in the local file system, such as " / mnt / s3fs". Through this path, users can access data in the object storage just like accessing local files.

[0054] Other mount configuration information may include mount options, such as whether to enable cache ("enable_cache=yes / no"), cache size ("cache_size=1024MB"), file system read and write permissions ("rw / ro" - read-write or read-only), etc. These options affect the performance and operation of the file system.

[0055] Furthermore, the first S3FS mounting parameter set represents relevant parameters used when the S3FS file system is initially mounted, and may include performance-related parameters, file system behavior parameters, and the like.

[0056] Among them, performance-related parameters may include:

[0057] (1) Cache parameters: These may include cache type (e.g., memory cache, disk cache), cache policy (e.g., least recently used - LRU, first in first out - FIFO, etc.), cache expiration time (e.g., the cache file is valid for 60 minutes), etc.

[0058] (2) Concurrency parameters: for example, the maximum number of concurrent connections ("max_concurrent_connections = 100"), which is used to control the number of connections that simultaneously access the file system to avoid system overload or performance degradation caused by too many connections; and limits on concurrent read and write operations (such as the maximum number of files that can be written simultaneously).

[0059] (3) Data transfer parameters: For example, the data block size ("block_size = 4096" bytes) used to set the data transfer unit in file read and write operations, as well as the transmission bandwidth limit ("max_transfer_rate = 10MBps"), etc. These parameters affect the data transfer efficiency of the file system.

[0060] Further, the file system behavior parameters may include:

[0061] (1) File permission parameters: For example, the default file owner ("uid = 1000"), group ("gid = 1000") permission settings, and the default permissions for new files ("mode = 0644") determine the user's access rights to files in the file system.

[0062] (2) File attribute parameters: For example, whether to support soft links of files ("allow_softlinks = yes / no"), the maximum size limit of files ("max_file_size = 10GB"), etc. These parameters define some basic attributes and operation rules of files in the file system.

[0063] Step S302, based on the user instruction, the mount key information set and the first S3FS mount parameter set, a target S3FS mount parameter set is obtained through DQN algorithm processing.

[0064] Among them, the DQN algorithm represents a Q-learning algorithm based on deep learning, which mainly combines value function approximation with neural network technology, and uses the target network and experience playback method to train the network. The core of the DQN algorithm is to maintain the Q function and use it to make decisions.

[0065] Specifically, based on the user instruction and the mount key information set, the first S3FS mount parameter set may be optimized by the DQN algorithm to obtain an optimized target S3FS mount parameter set.

[0066] The S3FS mount parameter optimization method provided in this embodiment optimizes the S3FS mount parameters through the DQN algorithm, and the adjustment direction of the mount parameters is clearer, the parameter optimization efficiency is higher, and it is simpler than the traditional optimization method relying on expert experience and manual debugging.

[0067] In this embodiment, a method for optimizing S3FS mount parameters is provided, which can be used as follows: Figure 1 The S3FS mount parameter optimization system 111 in the S3FS client 11 is shown. Figure 4 is a flow chart of a method for optimizing S3FS mount parameters according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps:

[0068] Step S401, obtain user instructions, a mount key information set and a first S3FS mount parameter set. Figure 3 Step S301 of the illustrated embodiment will not be described in detail here.

[0069] Step S402, based on the user instruction, the mount key information set and the first S3FS mount parameter set, a target S3FS mount parameter set is obtained through DQN algorithm processing.

[0070] Specifically, the above step S402 includes:

[0071] Step S4021: Based on the mount key information set, the S3FS file system is mounted using the first S3FS mount parameter set.

[0072] Specifically, according to the description of step S301, the mount key information set includes the core elements for connecting to the object storage service client, and is a key prerequisite for realizing communication between the S3FS file system and the object storage.

[0073] For example, the storage service endpoint (URL) specifies the network location of the object storage. Just like knowing the address of the destination, the S3FS client can find the corresponding storage service based on this information; the bucket name specifies the specific "container" where the data is stored, ensuring that subsequent operations can accurately locate the corresponding data storage space.

[0074] Furthermore, the local mount path in the mount key information set determines the location in the local file system through which to access the data in the object storage, allowing users to subsequently operate remotely stored data in a convenient way to access local files. For example, if the local mount path is set to " / mnt / s3fs", then when the user accesses the " / mnt / s3fs" directory locally, the actual operation is the content in the object storage. The Mount Options further refine the behavioral characteristics of the file system after mounting, such as whether to enable caching and the cache size, which will affect the subsequent performance of the file system.

[0075] Therefore, based on the storage service endpoint (URL) in the mount key information set, the S3FS client can initiate a network connection request with the object storage service, and carry authentication information such as the access key and secret key for identity authentication. If the authentication is successful, it means that the client has obtained the legal right to access the object storage, just like passing the access control check to enter a specific place, and at this time a communication link between the two is established.

[0076] Furthermore, according to the setting of the local mount path, the S3FS client can determine the mount point in the local file system, for example, mount the S3FS to the path " / mnt / s3fs" according to the setting. Then, according to the various parameters in the first S3FS mount parameter set, the mounted file system is configured in detail, including setting cache parameters (such as enabling memory cache, setting cache size, etc.), concurrency parameters (setting the maximum number of concurrent connections, etc.), data transmission parameters (determining data block size, etc.) and configuring parameters related to file system behavior (such as file permissions, etc.).

[0077] Step S4022: Based on the user instruction, control the operation of the S3FS file system and determine whether the performance of the S3FS file system is improved.

[0078] Specifically, the operation of the S3FS file system can be controlled under the control of user instructions, and it can be determined in real time whether the performance of the S3FS file system is improved.

[0079] In some optional implementations, the above step S4022 includes:

[0080] Step a1: Based on user instructions, control the operation of the S3FS file system to obtain an operation data set.

[0081] Step a2: determining multiple business performance indicator values ​​based on the operation data set.

[0082] Step a3: judging whether the performance of the S3FS file system is improved based on multiple business performance indicator values.

[0083] Specifically, under the control of user instructions, the S3FS file system mounted with the first S3FS mount parameter set is controlled to run, and a corresponding running data set is obtained during the running process, wherein the running data set may include time-related data (operation start event, operation end event, and time consumption of each sub-operation, etc.), data volume-related data (total amount of data read or written, size and number of data blocks, etc.), system resource usage data (CPU usage, memory usage, disk I / O status, etc.), network-related data (network bandwidth usage, network delay data, etc.), etc.

[0084] Furthermore, multiple indicators for measuring file system performance can be determined based on the obtained operating data set, which may include read and write performance indicators, response time indicators, resource utilization efficiency indicators, concurrency performance indicators, etc.

[0085] Among them, the read and write performance indicators may include:

[0086] (1) File reading speed: The average reading speed can be calculated by running the total amount of file reading operations recorded in the dataset and the operation time. For example, if it takes 5 seconds to read a 10MB file, the reading speed is 2MB / s;

[0087] (2) File write speed: This can be calculated based on the amount of data and time consumed in the write operation, and is used to measure the efficiency of writing data to the file system;

[0088] (3) Random read and write performance (for business scenarios with random read and write requirements): The performance of the file system in random read and write can be evaluated by simulating or recording actual random read and write operations (such as randomly reading and writing small blocks of data at different locations in a file) and calculating the average time taken for each random read and write operation and the number of random read and write operations that can be completed within a certain period of time.

[0089] Further, response time metrics may include:

[0090] (1) Average operation response time: The response time of various operations (read, write, query, etc.) can be comprehensively considered (i.e., the time interval from issuing an operation request to receiving feedback on the completion of the operation), and the average value can be calculated to reflect the overall response speed of the file system to user operations. For example, 10 different operations are performed, and the response time of each operation is recorded and the average value is calculated to obtain the average response time;

[0091] (2) Distribution of response time for specific operations: The response time distribution of certain key operations (such as frequently executed file read operations) can be analyzed. For example, the median and 90% quantile of the response time can be calculated to understand whether there are abnormal situations where the response time of some operations is too long.

[0092] Further, resource utilization efficiency indicators may include:

[0093] (1) CPU utilization: Based on the CPU utilization data recorded in the operation data set, the average CPU utilization can be calculated, that is, the percentage of the CPU time in use during the operation to the total time. This can be used to evaluate the CPU resource usage of file system operations and determine whether there is an over-consumption or under-utilization of CPU resources.

[0094] (2) Memory utilization: By analyzing memory usage data, we can calculate the average memory utilization and memory cache hit rate (cache hit count / total cache access count) and other indicators to determine whether memory resources are used reasonably and whether the cache mechanism is effective. For example, the higher the memory cache hit rate, the greater the role of the memory cache in improving file system performance.

[0095] (3) Disk I / O utilization (if disk operations are involved): You can view the disk I / O busyness, such as the disk I / O queue length, disk read and write operation saturation, and other indicators to analyze whether disk resources have become a bottleneck factor affecting file system performance.

[0096] Further, concurrent performance indicators may include:

[0097] (1) Maximum number of concurrent connections supported: Based on the concurrent operation data recorded in the running data set, we can analyze how many concurrent connections the file system can support without significantly degrading performance. This is very important for application scenarios where a large number of users access the file system at the same time.

[0098] (2) Concurrent operation throughput: Calculates the total number of operations that the file system can complete per unit time when multiple concurrent operations are performed simultaneously (such as the total number of read and write operations, the total amount of data transferred, etc.), and measures the overall processing capability of the file system in a concurrent environment.

[0099] Finally, combining multiple business performance indicator values ​​can determine whether the performance of the S3FS file system has improved.

[0100] Step S4023, when the performance of the S3FS file system is improved, the first S3FS mount parameter set is optimized using the DQN algorithm until the performance of the S3FS file system is not improved within a preset number of iterations, thereby obtaining a target S3FS mount parameter set.

[0101] Specifically, if the performance of the S3FS file system is improved, the DQN algorithm can be used to optimize the first S3FS mount parameter set until the performance of the S3FS file system is not significantly improved within a preset number of iterations, and a corresponding target S3FS mount parameter set is obtained.

[0102] In some optional implementations, the above step S4023 includes:

[0103] Step b1, when the performance of the S3FS file system is improved, the DQN algorithm is used to optimize the first S3FS mount parameter set and control the S3FS file system to stop running, thereby obtaining a second S3FS mount parameter set.

[0104] Step b2, using the second S3FS mount parameter set to mount the S3FS file system, returning to the step of controlling the operation of the S3FS file system, iterating repeatedly until the performance of the S3FS file system is not improved within a preset number of iterations, and obtaining a target S3FS mount parameter set.

[0105] Step b3: When the performance of the S3FS file system is not improved, return to the step of controlling the operation of the S3FS file system, and iterate repeatedly until the performance of the S3FS file system is not improved within a preset number of iterations, and obtain the target S3FS mount parameter set.

[0106] Specifically, if the performance of the S3FS file system is improved, the DQN algorithm may be used to optimize the first S3FS mount parameter set, and the S3FS file system may be controlled to stop running.

[0107] Among them, the DQN algorithm is shown in the following relationship:

[0108]

[0109] Where: S t A represents the state at time step t; t represents the action taken at time step t; ω represents the weight parameter of the neural network; Q(S t ,A t ,ω) means in state S t Next take action A t α represents the learning rate, which is a hyperparameter between 0 and 1 and is used to determine the proportion of newly acquired information in the update each time the Q value is updated; R t+1 represents the immediate reward (Reward) obtained at time step t+1; γ represents the discount factor (Discount factor), which is a hyperparameter between 0 and 1, used to weigh the importance of immediate rewards and future rewards. The larger γ is, the more important the future rewards are; the smaller γ is, the more important the immediate rewards are; Indicates that in state S t+1 The Q value estimate corresponding to the action with the largest Q value estimate among all possible actions is taken.

[0110] Further, according to the above relationship, the main idea of ​​the DQN algorithm is to select the best action among all possible actions in the next time step, and then use the Q value of this best action to update the action strategy target, that is, the Q value estimate of the current state-action pair can be adjusted according to the newly obtained reward and the expectation of future rewards, so that it is closer to the actual expected cumulative reward. This update method allows the intelligent agent to gradually find the optimal action strategy in different states by constantly interacting and learning with the environment.

[0111] When the DQN algorithm shown in the above relationship is used to optimize the S3FS mount parameters, Figure 5 As shown, first, from the state of the S3FS client default parameters S t Start, use the noisy-greedy strategy to select action A t , after completing the action, it will enter the new state S t+1 ; Next, check S t+1 In the middle action, extract the maximum value from the action as much as possible; finally, update the current action A with the above relationship t Q value, continuously iterated.

[0112] Further, the S3FS file system is mounted using the optimized second S3FS mount parameter set, and the S3FS file system mounted with the second S3FS mount parameter set is controlled to continue running, i.e., return to step S4022, and iterate repeatedly until the corresponding optimization result is outputted without obvious performance improvement for ten consecutive times, i.e., until the performance of the S3FS file system is not improved within the preset number of iterations, and the target S3FS mount parameter set is obtained.

[0113] In the process of optimizing S3FS mount parameters, the selection of Action is shown in Table 1 below:

[0114] Table 1

[0115]

[0116] Furthermore, the combination of multiple parameters constitutes multiple states, and the increase or decrease of the performance index of each state compared to the previous state is used as a reward. When selecting an action, the action with the largest Q value in the current state is used each time. When the performance index is stable for ten times, the iteration ends and the parameters mounted in the current state are output.

[0117] The S3FS mount parameter optimization method provided in this embodiment enables the system to mount and run the S3FS file system based on user instructions and key mount information. By continuously determining whether the performance is improved and continuing to optimize the parameters when the performance is improved, the performance of the S3FS file system can be significantly improved. At the same time, by using the DQN algorithm for parameter optimization, the need for manual intervention is reduced, solving the problem that traditional file system parameter configuration often requires a lot of manual experience and repeated experiments.

[0118] In this embodiment, a storage system is provided, such as Figure 2 As shown, the storage system 1 includes: an S3FS client 11 and an object storage service client 12.

[0119] Specifically, the object storage service client 12 is used to send a user instruction, a mount key information set, and a first S3FS mount parameter set to the S3FS client.

[0120] Furthermore, the S3FS client 11 includes an S3FS mount parameter optimization system 111 for executing the S3FS mount parameter optimization method provided by the above embodiment of the present invention.

[0121] In an optional real-time mode, the S3FS mount parameter optimization system 111 further includes: a business operation module 1111 , a business stop module 1112 , a mount parameter optimization module 1113 and an S3FS file system 1114 .

[0122] Specifically, the business operation module 1111 is used to run the S3FS file system 1114.

[0123] Furthermore, the business stopping module 1112 is used to stop the operation of the S3FS file system 1114.

[0124] Furthermore, the mount parameter optimization module 1113 is used to optimize the S3FS mount parameters of the S3FS file system 1114 to obtain a target S3FS mount parameter set.

[0125] The storage system provided in this example sends user instructions, a mount key information set, and a first S3FS mount parameter set to the S3FS client through the object storage service end. Further, based on these data, by executing the mount parameter optimization method on the S3FS client end, the interaction between the S3FS client and the object storage service end can be better adapted. Therefore, by implementing the present invention, the user only needs to input the user instructions, the mount key information set, and the first S3FS mount parameter set into the storage system to complete the automatic optimization of the S3FS mount parameters, which is simpler than the traditional optimization method that relies on expert experience and manual debugging.

[0126] In one example, a method for optimizing S3FS mount parameters based on a DQN algorithm is provided. Figure 6 As shown in the figure, based on the traditional mount parameter optimization method, the user only needs to write a test script to simulate the business to automatically analyze the optimal performance of the file system. Specifically, it includes:

[0127] (I) Optimizing S3FS mount parameters based on DQN algorithm

[0128] 1. Methods for obtaining key information

[0129] On the interface for executing the mount parameter optimization program, the user needs to provide the service start command, service stop command, key mount information, local path, etc. After providing the key information, the user can start optimizing the mount parameters by clicking the interface button, such as Figure 7 shown.

[0130] Furthermore, after the program is finished running, the recommended mounting method will be displayed in the user interface. The recommended mounting method is displayed in the form of a mounting command:

[0131] s3fs bucket001 / mnt / s3fs-test-o passwd_file= / root / .passwd-s3fs-o url=http: / / 192.168.149.141:7880-o use_path_request_style-o dbglevel=warn-oallow_other-o umask=0-o use_cache= / dev / shm-o enable_noobj_cache -o del_cache -o multipart_size=64 -o max_write=131072 -o big_writes -o max_stat_cache_size=100000 -o parallel_count=30 -o multireq_max=30 -o sigv2 -o kernel_cache-omax_background=1000.

[0132] 2. Program execution key process

[0133] After the user clicks the button to start optimization in the interface, the program starts to perform the optimization task through the DQN algorithm in the background. After multiple rounds of Action selection, the final result is output. The process is as follows: Figure 8 shown.

[0134] 3. Application of key algorithms

[0135] The S3FS mount parameter selection process is based on the DQN algorithm. The DQN algorithm is shown in the following equation:

[0136]

[0137] In the parameter optimization procedure of this example, Figure 5 As shown, the program will first start from the state S of the S3FS client default parameters t Start, use the noisy-greedy strategy to select action A t , after completing the action, it will enter the new state S t+1 ; Next, check S t+1 In the middle action, extract the maximum value from the action as much as possible; finally, update the current action A with the above relationship t Q value, continuously iterated.

[0138] Furthermore, in the mount parameter optimization method provided in this example, the selection of Action is shown in Table 1 above.

[0139] Furthermore, the combination of multiple parameters constitutes multiple states, and the increase or decrease of the performance index of each state compared to the previous state is used as a reward. When selecting an action, the action with the largest Q value in the current state is used each time. When the performance index is stable for five times, the iteration ends and the parameters mounted in the current state are output.

[0140] Furthermore, the S3FS mount parameter optimization method based on the DQN algorithm provided in this example can be applied to all S3SFS clients, and this example is applicable to storage products with object storage as the server.

[0141] The S3FS mount parameter optimization method based on the DQN algorithm provided in this example has the following beneficial effects:

[0142] (1) This example optimizes S3FS mount parameters, which is simpler than the traditional optimization method that relies on expert experience and manual debugging.

[0143] (2) In this example, the adjustment direction of the mounting parameters based on the DQN algorithm is clearer, and the efficiency of parameter optimization is higher.

[0144] The embodiment of the present invention also provides a computer device for executing the above Figure 3 or Figure 4 The S3FS mount parameter optimization method shown.

[0145] See also Fig. 9 , Fig. 9 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Fig. 9As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig. 9 A processor 10 is taken as an example.

[0146] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0147] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0148] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0149] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0150] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0151] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0152] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0153] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for optimizing S3FS mount parameters, characterized in that: An S3FS mount parameter optimization system for an S3FS client; the method comprises: Obtain user instructions, mount key information set and first S3FS mount parameter set; Based on the user instruction, the mount key information set and the first S3FS mount parameter set, a target S3FS mount parameter set is obtained through DQN algorithm processing.

2. The method according to claim 1, characterized in that The S3FS mount parameter optimization system includes an S3FS file system; based on the user instruction, the mount key information set and the first S3FS mount parameter set, a target S3FS mount parameter set is obtained through DQN algorithm processing, including: Based on the mount key information set, mounting the S3FS file system using the first S3FS mount parameter set; Based on the user instruction, controlling the operation of the S3FS file system and determining whether the performance of the S3FS file system is improved; When the performance of the S3FS file system is improved, the first S3FS mount parameter set is optimized using the DQN algorithm until the performance of the S3FS file system is not improved within a preset number of iterations, thereby obtaining the target S3FS mount parameter set.

3. The method according to claim 2, characterized in that Based on the user instruction, controlling the operation of the S3FS file system and determining whether the performance of the S3FS file system is improved includes: Based on the user instruction, control the operation of the S3FS file system to obtain an operation data set; Determine a plurality of service performance indicator values ​​based on the operation data set; Whether the performance of the S3FS file system is improved is determined based on the multiple business performance indicator values.

4. The method according to claim 2, characterized in that: When the performance of the file system is improved, the first S3FS mount parameter set is optimized using the DQN algorithm until the performance of the S3FS file system is not improved within a preset number of iterations, and the target S3FS mount parameter set is obtained, including: When the performance of the S3FS file system is improved, the first S3FS mount parameter set is optimized by using the DQN algorithm and the S3FS file system is controlled to stop running, thereby obtaining a second S3FS mount parameter set; The S3FS file system is mounted using the second S3FS mount parameter set, and the step of controlling the operation of the S3FS file system is returned to, and the process is repeated until the performance of the S3FS file system is not improved within a preset number of iterations, thereby obtaining the target S3FS mount parameter set.

5. The method according to claim 2, characterized in that: The method further comprises: When the performance of the S3FS file system is not improved, the process returns to the step of controlling the operation of the S3FS file system, and iterates repeatedly until the performance of the S3FS file system is not improved within a preset number of iterations, thereby obtaining the target S3FS mount parameter set.

6. A storage system, characterized in that: The system includes: an S3FS client and an object storage service client; The object storage service client is used to send a user instruction, a mount key information set and a first S3FS mount parameter set to the S3FS client; The S3FS client includes an S3FS mount parameter optimization system, which is used to execute the S3FS mount parameter optimization method according to any one of claims 1 to 5.

7. The system according to claim 6, characterized in that The S3FS mount parameter optimization system includes: a business operation module, a business stop module, a mount parameter optimization module and an S3FS file system; The business operation module is used to operate the S3FS file system; The service stopping module is used to stop the operation of the S3FS file system; The mount parameter optimization module is used to optimize the S3FS mount parameters of the S3FS file system to obtain a target S3FS mount parameter set.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the S3FS mount parameter optimization method according to any one of claims 1 to 5 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the S3FS mount parameter optimization method according to any one of claims 1 to 5.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the S3FS mount parameter optimization method according to any one of claims 1 to 5.

Images