IO performance optimization method and device, readable storage medium and program product

Abstract

The invention discloses an IO performance optimization method and device, a readable storage medium and a program product.The method comprises the steps that after it is detected that external storage equipment is mounted to a hard disk partition of target equipment, the type of a file system corresponding to the hard disk partition and the type of the external storage equipment are obtained; configuring IO parameters of a kernel space of the target equipment as IO parameters matched with the type of the file system and / or the type of the external storage equipment, wherein the kernel space comprises at least one of a general block equipment layer, an IO scheduling layer and a block equipment driving layer; and optimizing the IO performance of the target equipment based on the configured IO parameters.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to an IO performance optimization method and apparatus, a readable storage medium, and a program product. Background Technology

[0002] With technological advancements and improved hardware configurations, I / O performance, including I / O speed, has also increased significantly. While I / O speed can be improved through hardware upgrades, these are costly. Currently, software-based I / O performance optimization is being explored to fully leverage the capabilities of individual hardware components. For example, optimization can be achieved through applications on devices with USB interfaces, such as computer hosts that have external storage. Since the application sits at the top of the I / O stack, it can adjust I / O modes via system calls, such as sequential / random or synchronous / asynchronous. Alternatively, purchasing a commercially available file system can improve the device's file system I / O performance.

[0003] While the methods described above can improve I / O performance to some extent, overall I / O performance remains low, with significant room for improvement. How to further and effectively increase I / O speed and optimize I / O performance is a key technical challenge that needs to be addressed. Summary of the Invention

[0004] The purpose of this application is to provide an IO performance optimization method and apparatus, a readable storage medium, and a program product to solve the problem of low IO performance.

[0005] To solve the above-mentioned technical problems, this specification is implemented as follows: Firstly, a method for optimizing I / O performance is provided, including: After detecting the hard disk partition of the external storage device mounted to the target device, obtain the file system type corresponding to the hard disk partition and the type of the external storage device; The IO parameters of the target device kernel space are configured to match the type of the file system and / or the type of the external storage device, wherein the kernel space includes at least one of a general block device layer, an IO scheduling layer, and a block device driver layer; Based on the configured IO parameters, the IO performance of the target device is optimized.

[0006] Optionally, the I / O parameters of the target device kernel space are configured to match the type of the file system and / or the type of the external storage device, including at least one of the following: Configure the maximum amount of data transferred in a single I / O request submitted by the block device driver layer to match the maximum amount of data transferred in accordance with the type of the file system and / or the type of the external storage device; Configure the amount of prefetched data in the general block device layer to match the type of the external storage device; Configure the amount of data to be read beforehand to match the maximum amount of data to be transmitted.

[0007] Optionally, it also includes: When performing data prefetching on the external storage device in response to an I / O request for a read operation, the data prefetching is performed using multiple threads.

[0008] Optionally, the data prefetching is performed using multiple threads, including: Determine the method of data pre-reading; If the data prefetching method is synchronous prefetching, then in response to the IO request of the read operation, the data is cached from the external storage device by the first thread and copied to the user space by the second thread; If the data prefetching method is asynchronous prefetching, then in response to the IO request of the read operation, the third thread checks whether the data on the external storage device is continuous and the fourth thread submits the IO request.

[0009] Optionally, it also includes: If the data on the external storage device is checked to be continuous, the BIO size of the block device driver layer is configured by merging the BIOs corresponding to the IO requests of multiple consecutive read operations into the same BIO.

[0010] Optionally, the I / O parameters of the target device kernel space are configured to match the type of the file system and / or the type of the external storage device, including at least one of the following: Configure the scheduling policy of the IO scheduling layer to match the type of the external storage device; When multiple IO requests for reading multiple files are detected simultaneously, the front-end merging parameter of the IO scheduling layer is configured to perform front-end merging on the multiple IO requests; When multiple users simultaneously submit I / O requests to the external storage device, the maximum read / write response time of the I / O scheduling layer is reduced. When the frequency of read operations and write operations corresponding to IO requests is inconsistent within a preset time period, the write hunger parameter of the IO scheduling layer is configured to match the ratio between the number of read operations and the number of write operations within the preset time period.

[0011] Optionally, it also includes: When writing data to the external storage device in response to an IO request for a write operation, the request data volume of the general block device layer is configured to be synchronized with the write-back data volume threshold by modifying the length of the request queue or the write-back data volume threshold. The requested data volume is determined based on the length of the request queue, the number of request queues, and the maximum amount of data transmitted by a single IO request. The request queue is used to place IO requests for write operations and IO requests for write-back operations corresponding to the write operations. The cached data volume corresponding to the IO requests for write operations is written back to the external storage device when it reaches the write-back data volume threshold.

[0012] Optionally, it also includes: When writing data to the external storage device in response to an IO request for a write operation of target file data, an expected value and an upper limit value of physical blocks are pre-allocated to store the target file data, wherein the expected value is less than the upper limit value; After the physical block of the expected value is filled with the target file data, if an IO request for writing the target file data is received, the expected value is increased by a preset exponential level. The target file data is written based on the increased expected value of the physical blocks. Repeat the steps from receiving an IO request for a write operation on the target file data to writing the target file data to a physical block based on the increased expected value, until the increased expected value meets the preset condition of the upper limit value or no IO request for a write operation on the target file data is received.

[0013] Secondly, an I / O performance optimization device is provided, comprising: The acquisition module, after detecting the hard disk partition of the external storage device mounted to the target device, acquires the file system type corresponding to the hard disk partition and the type of the external storage device; The configuration module configures the IO parameters of the target device kernel space to match the type of the file system and / or the type of the external storage device. The kernel space includes at least one of a general block device layer, an IO scheduling layer, and a block device driver layer. The optimization module optimizes the I / O performance of the target device based on the configured I / O parameters.

[0014] Thirdly, an I / O performance optimization apparatus is provided, including a processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the method described in the first aspect.

[0015] Fourthly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0016] Fifthly, a computer program product is provided, comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform the steps of the method described in the first aspect.

[0017] In this embodiment, after detecting that an external storage device is mounted to a hard disk partition on the target device, the type of the file system corresponding to the hard disk partition and the type of the external storage device are obtained; the IO parameters of the target device's kernel space are configured to match the type of the file system and / or the type of the external storage device, wherein the kernel space includes at least one of a general block device layer, an IO scheduling layer, and a block device driver layer; based on the configured IO parameters, the IO performance of the target device is optimized, thereby optimizing the IO process of the kernel space from a software perspective. When performing corresponding read and write operations, the utilization of the device's CPU and the read and write speed of the external storage device are improved, achieving optimal IO performance in various scenarios and saving optimization costs. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic diagram of the IO process architecture of the target device in an embodiment of this application.

[0019] Figure 2 This is a flowchart illustrating the IO performance optimization method according to an embodiment of this application.

[0020] Figure 3 This is a schematic diagram illustrating an application scenario of the IO performance optimization method according to an embodiment of this application.

[0021] Figure 4 This is a schematic diagram of the pre-read process of the IO performance optimization method according to an embodiment of this application.

[0022] Figure 5 This is a schematic diagram of the parameter configuration process of the IO scheduling layer in an embodiment of this application.

[0023] Figure 6 This is a schematic diagram of the parameter configuration process of the general block device layer in an embodiment of this application.

[0024] Figure 7 This is a schematic diagram of the physical block pre-allocation process according to an embodiment of this application.

[0025] Figure 8 This is an example diagram of physical block pre-allocation at different stages in the embodiments of this application.

[0026] Figure 9 This is a structural block diagram of the IO performance optimization device according to an embodiment of this application. Detailed Implementation

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. The drawing numbers in this application are only used to distinguish the various steps in the solution and are not used to limit the execution order of the various steps. The specific execution order is subject to the description in the specification.

[0028] To address the problems existing in the prior art, embodiments of this application provide an IO performance optimization method and apparatus, a readable storage medium, and a program product, so as to further and effectively improve IO speed and optimize IO performance from a software perspective.

[0029] Figure 1 This is a schematic diagram of the IO process architecture of the target device in an embodiment of this application, such as... Figure 1 As shown, the IO process architecture includes the application 12 in user space 100, and the virtual file system in kernel space 200, the specific file system 22 (e.g., NTFS / FAT32), the general block device layer 23, the IO scheduling layer 24, the block device driver 25, and the hard disk 26.

[0030] This application's embodiments optimize I / O performance from a software perspective, fully leveraging the hardware's performance and improving its read / write speed.

[0031] refer to Figure 2 The IO performance optimization method of this application embodiment includes steps 102 to 106. Combined with... Figure 3 The configuration operations in steps 102 to 104 can be performed by the configuration module 32.

[0032] Step 102: After detecting the hard disk partition of the external storage device mounted to the target device, obtain the file system type corresponding to the hard disk partition and the type of the external storage device.

[0033] Target devices include computer hosts with USB interfaces, and external storage devices 40 include hard drives, solid-state drives (SSDs), and USB flash drives. External storage devices 40 can be plugged into and unplugged from the target devices via USB interfaces. The block device driver detects changes in port levels, wakes up the relevant kernel process, and checks whether a new external storage device 40 has been inserted or removed (step 302). If a plugging or unplugging process is detected, it identifies which port (including HUB expansion cases) the action occurred on and then sends a hotplug message to the user space. The hotplug process receives the message, identifies the message type, and executes the corresponding script. The message type for plugging or unplugging external storage devices 40 is a block type message, so after receiving the hotplug message, it executes the scripts in the corresponding directory in sequence.

[0034] Then, a mount point, i.e., a hard disk partition mount directory, is created, and the mounting is performed (step 304). After the external storage device 40 is successfully mounted, the mount information of the corresponding hard disk partition is added. The mount information records the file system type and external storage device type corresponding to the external storage device 40. Thus, the configuration module 32 can obtain the corresponding file system type and external storage device type from the mount information.

[0035] Step 104: Configure the IO parameters of the target device kernel space to match the type of the file system and / or the type of the external storage device. The kernel space includes at least one of a general block device layer, an IO scheduling layer, and a block device driver layer.

[0036] In this embodiment, the general block device layer, IO scheduling layer and / or block device driver layer can be optimized. Specifically, the configuration module 32 configures the corresponding matching IO parameters according to the file system type and external storage device type in the mount information.

[0037] In one embodiment, configuring the IO parameters of the target device kernel space to match the type of the file system and / or the type of the external storage device includes at least one of the following: configuring the maximum amount of data transferred in a single IO request submitted by the block device driver layer to match the maximum amount of data transferred in the file system and / or the type of the external storage device; configuring the amount of data to be read in the general block device layer to match the amount of data to be read in the external storage device; and configuring the amount of data to be read in the external storage device to match the maximum amount of data transferred.

[0038] 1. Optimization of maximum data transmission volume The amount of data submitted in a single IO request by the block device driver layer, i.e. the amount of data in a read operation or a write operation, cannot exceed the maximum data transfer amount max_sectors_kb. The default value of the maximum data transfer amount is initialized by the block device driver layer driver, and can be configured or adjusted by modifying the kernel code through configuration module 32.

[0039] The configuration module 32 can dynamically adjust the maximum data transfer parameter based on the file system type. First, the configuration module 32 can obtain the file system type of the hard disk partition mounted on the external storage device 40 from the mount information, and then adjust the corresponding maximum data transfer based on this file system type. The default value is 128. Specifically, when the file system is NTFS-3G, max_sector_kb is adjusted to 512; when the file system is NTFS3, max_sector_kb is adjusted to 2048; when the file system is FAT32, max_sector_kb is adjusted to 512; and when the file system is exFAT, max_sector_kb is adjusted to 2048.

[0040] For example, in an NTFS3 file system, changing max_sectors_kb to 512K can increase the data read speed from 215MB / s to 240MB / s, while the data write speed increases from 100MB / s to 140MB / s.

[0041] It is important to note that for the FAT32 file system, the remaining space on the external storage device disk has a significant impact on write performance. Table 1 below shows the impact of different max_sector_kb configurations on the write performance of various file systems when the remaining space on the external storage device disk is relatively sufficient.

[0042] Table 1

[0043] By configuring the maximum data transfer volume parameter to match the file system type, the performance of external storage device 40 can be better utilized. Increasing this parameter can increase the amount of data committed to the driver at one time. Increasing the maximum data transfer volume can improve the IO rate during large file transfers, and better utilize the throughput of external storage device 40.

[0044] The parameter `max_sector_kb` behaves differently on different types of external storage devices 40, such as HDDs and SSDs. The configuration module 32 can obtain the type of external storage device 40 from the mount information and then adjust the corresponding maximum data transfer volume based on the type of external storage device 40. This parameter represents the maximum amount of data submitted to the driver at one time. Testing shows that the optimal value for this parameter is 512K on SSDs, but 120K on mechanical hard drives. Special handling can be applied to different types of external storage devices. There is a dynamically adjustable parameter for different types of disks or SSDs with different performance levels.

[0045] Specifically, increasing the maximum data transfer capacity (max_sector_kb) for the same external storage device can fully utilize the hardware performance of external storage devices with different performance levels, and the write speed can also be significantly improved.

[0046] 2. Optimization of pre-read data volume The target device's operating system typically uses a default read-ahead size of 128K. However, the read-ahead_kb value, which allows the file system to reach its maximum speed, varies depending on the type of external storage device 40. This is particularly crucial for reading large files, as accurately predicting the application's sequential read behavior can fully utilize the performance of the external storage device 40. The configuration module 32 can obtain the type of external storage device 40 from the mount information and then adjust the corresponding read-ahead data amount based on the type of external storage device 40. For example, in the NTFS3 file system, changing read_ahead_kb from 128K to 1024K can increase the read speed on an SSD from 215MB / s to 260MB / s.

[0047] 3. Matching the maximum data transmission volume with the pre-read data volume. The size of the prefetch data is not always better the larger it is. With a maximum transfer data size of 120K (max_sectors_kb), reducing the prefetch data size from 128K to 120K results in roughly the same speed improvement as increasing it to 1024K. When the original prefetch data size is 128K, due to the 120K maximum transfer data size limit, an IO request needs to be split into 120K + 8K, i.e., split into two IO requests. Reducing the prefetch data size to 120K eliminates the need for splitting, thus reducing the number of IO requests. Furthermore, when the external storage device is an SSD, a prefetch data size of 120K results in higher read speeds. Taking NTFS3 as an example, the test data on the correspondence between the two parameters, read_ahead_kb and max_sectors_kb, are shown in Table 2. Table 2 shows the impact of the NTFS3 read-ahead data size and the maximum data transfer size for a single IO request on read performance.

[0048] Table 2

[0049] As can be seen from Table 2, changing read_ahead_kb and max_sector_kb to matching values ​​will significantly improve IO performance.

[0050] The read speeds of various file systems were tested on the external storage device of HDD, as shown in Table 3. Table 3 shows the impact of matching optimization of parameters ead_ahead_kb and max_sector_kb on the read and write performance of each type of file system.

[0051]

[0052] Based on the solution provided in the above embodiments, the method may optionally further include: performing the data pre-reading of the external storage device through multiple threads when performing data pre-reading in response to an IO request for a read operation.

[0053] Data prefetching involves caching not only the data to be read from the external storage device (e.g., a dirty page) to the kernel upon receiving a single I / O request, but also caching additional dirty pages simultaneously. The multiple steps of the prefetching process are performed sequentially, which can lead to excessive time consumption. This embodiment utilizes a multi-threaded mechanism to execute the corresponding tasks in parallel, optimizing the prefetching time and improving I / O read performance.

[0054] The prefetching process includes different types, such as synchronous prefetching and asynchronous prefetching. In synchronous prefetching, data can only be copied to user space after all dirty pages included in the prefetched data volume are cached in the kernel. Typically, the prefetched data volume is larger than the amount of data the application expects to read. In asynchronous prefetching, the dirty pages that triggered the system call can only be copied to user space after the I / O request has been submitted. Before submitting the I / O request, it is also necessary to check whether the data stored on the external storage device is contiguous, which also incurs time consumption.

[0055] Therefore, in one embodiment, the data prefetching is performed by multiple threads, including: determining the data prefetching method; if the data prefetching method is synchronous prefetching, then in response to the IO request of the read operation, data is cached from the external storage device by a first thread and the cached data is copied to user space by a second thread; if the data prefetching method is asynchronous prefetching, then in response to the IO request of the read operation, the data on the external storage device is checked for continuity by a third thread and the IO request is submitted by a fourth thread.

[0056] Combination Figure 3 When a request is made to read data from external storage device 40, the prefetch module 34 configures multiple threads to perform the corresponding prefetch (step 306), thereby caching the dirty pages prefetched into the memory of hard disk 26.

[0057] For information on different types of prefetching processes in multithreading, please refer to [link / reference]. Figure 4 ,like Figure 4 As shown, for synchronous prefetching, the prefetch module 34 allocates thread 1 from the thread pool to cache pages (step 402), and allocates thread 2 to copy the cached pages to user space (i.e., user space) (step 404). For asynchronous prefetching, in addition to threads 1 and 2, the prefetch module 34 also allocates thread 3 from the thread pool to check whether the hard disk of the external storage device 40 is a contiguous storage space (step 406), and submits an IO request through thread 4 (step 408). The steps of caching pages and copying cached pages to user space involved in asynchronous prefetching are executed by the corresponding threads 1 and 2. The above threads are different threads.

[0058] By employing a multi-threaded mechanism, when reading data from an external storage device, the system first checks if the data on the external storage device's hard drive is already cached in the page cache. If the corresponding page is found, the pre-fetch logic is entered, and the data is directly copied to the user space. This multi-threading mechanism separates the caching of pages and the copying of pages to user space in synchronous pre-fetching—one thread handles caching, and another handles copying—allowing for simultaneous caching and copying, reducing latency and improving I / O performance. Similarly, in asynchronous pre-fetching, multi-threading separates the checking of the external storage device's hard drive's storage space continuity, submitting I / O requests, caching pages, and copying pages to user space, further reducing latency and improving I / O performance.

[0059] Optionally, the method further includes: if the data on the external storage device is checked to be continuous, then configuring the BIO size of the block device driver layer by merging the BIOs corresponding to the IO requests of multiple consecutive read operations into the same BIO.

[0060] BIO represents the data structure for I / O requests. It contains various information describing the I / O request, such as the data buffer, starting sector, and length. When an application or file system needs to perform read or write operations, it encapsulates the corresponding I / O request into a BIO structure and submits it to the block device layer for processing. The size of the BIO can be specified by the file system. For example, when the amount of data to be pre-read is 4MB, the file system will generate four BIOs of size 1MB each.

[0061] In scenarios where the pages to be read are not stored contiguously on the hard drive, they will be split into multiple BIOs. If contiguous storage is detected on the external storage device when initiating a read operation, the BIOs corresponding to multiple consecutive read operations can be merged into a single BIO. By adjusting the size of the BIO, the number of I / O requests can be reduced, thereby improving the I / O rate.

[0062] This application can increase the size of a single generated BIO, reduce the number of IO requests, and improve IO speed in block device driver layer scenarios involving large file read and write operations.

[0063] Based on the solution provided in the above embodiments, optionally, in step 104, configuring the IO parameters of the target device kernel space to match the type of the file system and / or the type of the external storage device includes at least one of the following: configuring the scheduling policy of the IO scheduling layer to match the type of the external storage device; when multiple IO requests for reading multiple files are detected simultaneously, configuring the front-end merging parameter of the IO scheduling layer to perform front-end merging on the multiple IO requests; when multiple users submit IO requests to the external storage device simultaneously, configuring the maximum read / write response time of the IO scheduling layer to be reduced; when the frequency of read operations and write operations corresponding to IO requests is inconsistent within a preset time period, configuring the write hunger parameter of the IO scheduling layer to match the ratio between the number of read operations and the number of write operations within the preset time period.

[0064] For reading and writing single large files, for performance reasons, files are stored in contiguous physical space, and there are few scenarios requiring merging and scheduling. Therefore, the scheduler has not played its role. However, in scenarios such as reading multiple files, simultaneous access by multiple users, and physical fragmentation caused by long-term use, the scheduler can play an important role.

[0065] In this embodiment of the application, the scheduling process can be dynamically adjusted by the configuration module 32 for different types of external storage devices. For example, for SSDs, whose storage medium is mainly flash memory chips, it is better to configure noop scheduling; for non-SSD mechanical hard drives, it is better to configure deadline scheduling.

[0066] Furthermore, it can be dynamically adjusted for different processing scenarios, such as scenarios with multiple users, multiple files, and inconsistent frequencies of read and write operations.

[0067] When dealing with multi-user scenarios, the number of users can be recorded through configuration module 32. When the number of users submitting IO requests at the same time is too large, the maximum read and write response time in the IO scheduling process can be shortened. read_expire represents the maximum response time for read operations, and write_expire represents the maximum response time for write operations. Matching parameter adjustments can be achieved by reducing read_expire and write_expire.

[0068] When dealing with multiple files, file data can be recorded through configuration module 32. When the number of files read at the same time is too large, the front_merges parameter of the IO scheduling layer can be configured to perform front-end merging of multiple IO requests.

[0069] When faced with scenarios where the frequency of read and write operations is inconsistent, the write starvation parameter `write_starved` in the I / O scheduling layer can be dynamically adjusted through configuration module 32. This parameter indicates how many read operations are executed in the kernel before considering a write operation. Specifically, a flag can be set to record the number of read and write operations within a preset time period, determining the ratio of read to write operations and dynamically adjusting the `write_starved` parameter accordingly. When the ratio increases, the `write_starved` parameter can be increased; when the ratio decreases, the `write_starved` parameter can be decreased.

[0070] By dynamically adjusting some IO parameters in various scenarios, the IO performance can be optimized for each scenario.

[0071] The following is combined with Figure 5 The parameter configuration process of the IO scheduling layer in this application embodiment is described, such as... Figure 5 As shown, it includes the following steps: Step 502: Mount the external storage device to the device's hard disk partition; Step 504: Write the hard disk partition information, including the corresponding file system type and the type of external storage device; Step 506: Determine whether the external storage device is an SSD. If it is, proceed to step 508; otherwise, proceed to step 510. Step 508: Configure the scheduling policy as noop scheduling; Step 510: Configure the scheduling policy as deadline scheduling; Step 512: Determine if it is a multi-file read. If yes, proceed to step 514; otherwise, proceed to step 516. Step 514, adjust the front-end merging parameter front_merges; Step 516: Determine if it is a multi-user read; if yes, proceed to step 518; otherwise, end. Step 518: Adjust the parameters read_expire, write_expire, and write_starved.

[0072] In this embodiment, by dynamically adjusting the configuration in the IO process, such as the scheduling mechanism, front-end merging parameters, maximum read / write response time, and read / write response frequency, the role of the IO scheduler can be fully utilized.

[0073] Optionally, the method further includes: when writing data to the external storage device in response to an IO request for a write operation, configuring the request data volume of the general block device layer to be synchronized with the write-back data volume threshold by modifying the length of the request queue or the write-back data volume threshold; wherein the request data volume is determined based on the length of the request queue, the number of request queues, and the maximum amount of data transferred by a single IO request, the request queue is used to place the IO request for the write operation and the IO request for the write operation corresponding to the write operation, and the cached data volume corresponding to the IO request for the write operation is written back to the external storage device when the write-back data volume threshold is reached.

[0074] This embodiment configures the parameters of the general block device layer through configuration module 32. When there is an I / O request for a write operation, the I / O request is added to the request queue. Due to the caching effect of the page cache, the write operation is actually delayed. When the data in the page cache is newer than the data in the background storage, the data is called a dirty page. Dirty pages accumulated in memory must eventually be written back to disk. The write-back thread is a mechanism in the kernel responsible for writing dirty pages in the cache back to the underlying storage device. During writing, the write-back thread continuously submits I / O requests for write-back operations, causing the request queue to fill up. Only after the request queue is empty will a new write I / O operation be submitted again. Processing the I / O requests already submitted in the request queue will generate high time consumption, causing each write I / O request to wait in the request queue for a long time.

[0075] Therefore, by modifying the parameter nr_reques, which represents the length of the request queue, for example by reducing the length of the request queue, the data writing speed can be significantly improved.

[0076] In addition, a mount parameter in the file system, namely the writeback data threshold `writeback_boundary`, indicates that once the number of dirty pages in the cache reaches the number configured for this mount parameter, a forced flush and reclamation are required. The kernel uses a request queue mechanism; when a writeback is needed, the task is submitted to the request queue. The advantage of using a request queue is that it saves resources, allowing multiple tasks to share resources as much as possible, thus avoiding excessive resource waste.

[0077] If the length of the request queue is changed to 32, and the maximum data transfer amount for each IO request is 120K, then the amount of request data in the request queue will be close to 4M.

[0078] The requested data volume is calculated as follows: Requested data volume = d length of request queue × number of request queues × maximum data volume of a single IO request.

[0079] When the requested data volume is close to the write-back data volume threshold, the IO rate can be improved. In this embodiment, by synchronizing the requested data volume and the write-back data volume threshold of the request queue, the IO rate can be effectively improved. When one parameter, the write-back data volume threshold or the length of the request queue, changes, the other parameter is adjusted synchronously. Here, synchronization can mean that the requested data volume and the write-back data volume threshold are equal or approximately equal.

[0080] refer to Figure 6 For example, when the length of the configuration request queue nr_reques is reduced to improve the data writing speed, the writeback data volume threshold writeback_boundary, which is synchronized with the request data volume, can be calculated based on the configured parameter nr_reques (step 602).

[0081] Accordingly, when the writeback data volume threshold writeback_boundary changes, the request queue length nr_reques corresponding to the request data volume synchronized with the writeback data volume threshold can be calculated based on the configured parameter writeback_boundary (step 604).

[0082] To ensure data storage is as continuous as possible, effectively increase IO throughput, reduce the number of IO requests, and decrease disk seek time, the method may optionally further include: when writing data to the external storage device in response to an IO request for a write operation of target file data, pre-allocating an expected value and an upper limit value for physical blocks used to store the target file data, wherein the expected value is less than the upper limit value; after the physical block containing the expected value is filled with the target file data, if an IO request for a write operation of the target file data is received, increasing the expected value by a preset exponential level; writing the target file data based on the physical block containing the increased expected value; repeating the steps from receiving an IO request for a write operation of the target file data to writing the target file data based on the physical block containing the increased expected value, until the increased expected value meets the preset condition of the upper limit value or no IO request for a write operation of the target file data is received.

[0083] Combination Figure 3 In this embodiment of the application, the pre-allocation module 36 pre-allocates large blocks of physical space for the file in advance by pre-allocating physical blocks.

[0084] Each time file data is written, actual physical blocks need to be allocated at the underlying level. This allocation process is relatively time-consuming. Secondly, due to underlying fragmentation, contiguous file data at the upper level may not be stored in contiguous physical space. Storing files in contiguous physical space effectively increases I / O throughput, reduces the number of I / O operations, and also reduces disk seek time. This application uses a pre-allocation mechanism to allocate large blocks of physical space for files in advance; this is a dynamic pre-allocation.

[0085] If the upper-layer application knows the file size, it can allocate a large amount of physical space at once. If the file size is unknown, it can pre-allocate the expected number of physical blocks to store the file data. After the file data is written to the expected number of physical blocks, the expected value is adaptively adjusted based on whether the physical blocks are full.

[0086] If the physical block containing the desired data is filled with the target file data, it means the amount of file data to be written is greater than the desired value. In this case, the desired value can be increased exponentially, i.e., the pre-allocated physical block size for writing file data is updated. This process is repeated until a preset condition is met, at which point the desired value is no longer increased. The preset condition is, for example, the upper limit of the physical block size used to store file data. For instance, the desired size might be 64KB (the size of one physical block), and the upper limit might be 976MB, depending on the file system type.

[0087] The following is based on Figure 7 Taking an example, the physical block pre-allocation process of an embodiment of this application will be described. Figure 7 As shown, it includes the following steps: Step 702: Configure the expected value size through the pre-allocation module 36, which represents the expected size of the physical blocks to be pre-allocated, and configure the maximum value max, which represents the upper limit size of the pre-allocated physical blocks. Step 704: Increase the expected value size by 2 as a preset exponent, that is, round up to the power of 2, and assign the increased expected value to the new expected value newsize; Step 706: Determine whether the new expected value meets the preset conditions, such as whether it is less than half of the upper limit value. If yes, proceed to step 708; otherwise, proceed to step 710. Step 708: Increase the new expected value using the formula newsize = newsize * 2; Step 710: Set the new expected value as the upper limit value; Step 712, return the new expected value.

[0088] The returned new expected value serves as the physical block size for the I / O request to store file data. Alternatively, physical block pre-allocation stops if no subsequent I / O requests for file data are received.

[0089] Combination Figure 8 This describes the process of pre-allocating physical blocks. For example... Figure 8 As shown, after receiving the IO request for writing file data in stage 1, one physical block block1 is pre-allocated; after receiving the IO request for writing file data in stage 2, four physical blocks block1 to block4 are pre-allocated exponentially at a size four times that of the physical blocks in stage 1. The physical block block2 marked in red indicates the node that triggered the new pre-allocation action.

[0090] Similarly, after receiving the IO request for writing file data in stage 3, 16 physical blocks (block1 to block16) are pre-allocated at an exponential rate of 4 times the physical block size of stage 2. The physical block (block5) marked in red represents the node that triggers the new pre-allocation action.

[0091] By exponentially increasing the expected value of pre-allocated physical blocks, the file system's write efficiency can be optimized. Pre-allocating large blocks of physical space for files effectively increases I / O throughput, reduces the number of I / O operations, and also reduces disk seek time.

[0092] Step 106: Optimize the IO performance of the target device based on the configured IO parameters.

[0093] After configuring the relevant IO parameters through the above steps, the IO performance of the general block device layer, IO scheduling layer, and / or block device driver layer can be optimized when performing corresponding read and write operations, thereby improving the utilization of the device CPU and the read and write speed of the external storage device. By dynamically adjusting the corresponding IO parameters under various scenarios, the IO performance can be optimized for each scenario.

[0094] In this embodiment, after detecting that an external storage device is mounted to a hard disk partition on the target device, the type of the file system corresponding to the hard disk partition and the type of the external storage device are obtained; the IO parameters of the target device's kernel space are configured to match the type of the file system and / or the type of the external storage device, wherein the kernel space includes at least one of a general block device layer, an IO scheduling layer, and a block device driver layer; based on the configured IO parameters, the IO performance of the target device is optimized, thereby optimizing the IO process of the kernel space from a software perspective. When performing corresponding read and write operations, the utilization of the device's CPU and the read and write speed of the external storage device are improved, achieving optimal IO performance in various scenarios and saving optimization costs.

[0095] Optionally, embodiments of this application also provide an IO performance optimization device, including: The acquisition module, after detecting the hard disk partition of the external storage device mounted to the target device, acquires the file system type corresponding to the hard disk partition and the type of the external storage device; The configuration module configures the IO parameters of the target device kernel space to match the type of the file system and / or the type of the external storage device. The kernel space includes at least one of a general block device layer, an IO scheduling layer, and a block device driver layer. The optimization module optimizes the I / O performance of the target device based on the configured I / O parameters.

[0096] It is understood that the IO performance optimization device provided in the embodiments of this specification can correspondingly implement... Figures 1 to 8 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0097] Optionally, such as Figure 9 As shown, this application embodiment also provides an IO performance optimization device 2000, including a processor 2400 and a memory 2200. The memory 2200 stores a program or instructions that can run on the processor 2400. When the program or instructions are executed by the processor 2400, they implement the various steps of the above-described IO performance optimization method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0098] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of any of the above-described IO performance optimization method embodiments and achieve the same technical effect. To avoid repetition, further details are omitted here. The readable storage medium includes computer-readable storage media, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0099] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. The computer program is operable to cause a computer to execute various processes of any of the above-described IO performance optimization method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0100] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0101] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0102] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An I / O performance optimization method, characterized in that, include: After detecting the hard disk partition of the external storage device mounted to the target device, obtain the file system type corresponding to the hard disk partition and the type of the external storage device; The IO parameters of the target device kernel space are configured to match the type of the file system and / or the type of the external storage device, wherein the kernel space includes at least one of a general block device layer, an IO scheduling layer, and a block device driver layer; Based on the configured IO parameters, the IO performance of the target device is optimized.

2. The method according to claim 1, characterized in that, Configure the I / O parameters of the target device kernel space to match the type of the file system and / or the type of the external storage device, including at least one of the following: Configure the maximum amount of data transferred in a single I / O request submitted by the block device driver layer to match the maximum amount of data transferred in accordance with the type of the file system and / or the type of the external storage device; Configure the amount of prefetched data in the general block device layer to match the type of the external storage device; Configure the amount of data to be read beforehand to match the maximum amount of data to be transmitted.

3. The method according to claim 1, characterized in that, Also includes: When performing data prefetching on the external storage device in response to an I / O request for a read operation, the data prefetching is performed using multiple threads.

4. The method according to claim 3, characterized in that, The data prefetching is performed using multiple threads, including: Determine the method of data pre-reading; If the data prefetching method is synchronous prefetching, then in response to the IO request of the read operation, the data is cached from the external storage device by the first thread and copied to the user space by the second thread; If the data prefetching method is asynchronous prefetching, then in response to the IO request of the read operation, the third thread checks whether the data on the external storage device is continuous and the fourth thread submits the IO request.

5. The method according to claim 1, characterized in that, Configure the I / O parameters of the target device kernel space to match the type of the file system and / or the type of the external storage device, including at least one of the following: Configure the scheduling policy of the IO scheduling layer to match the type of the external storage device; When multiple IO requests for reading multiple files are detected simultaneously, the front-end merging parameter of the IO scheduling layer is configured to perform front-end merging on the multiple IO requests; When multiple users simultaneously submit I / O requests to the external storage device, the maximum read / write response time of the I / O scheduling layer is reduced. When the frequency of read operations and write operations corresponding to IO requests is inconsistent within a preset time period, the write hunger parameter of the IO scheduling layer is configured to match the ratio between the number of read operations and the number of write operations within the preset time period.

6. The method according to claim 1, characterized in that, Also includes: When writing data to the external storage device in response to an IO request for a write operation, the request data volume of the general block device layer is configured to be synchronized with the write-back data volume threshold by modifying the length of the request queue or the write-back data volume threshold. The requested data volume is determined based on the length of the request queue, the number of request queues, and the maximum amount of data transmitted by a single IO request. The request queue is used to place IO requests for write operations and IO requests for write-back operations corresponding to the write operations. The cached data volume corresponding to the IO requests for write operations is written back to the external storage device when it reaches the write-back data volume threshold.

7. The method according to claim 1, characterized in that, Also includes: When writing data to the external storage device in response to an IO request for a write operation of target file data, an expected value and an upper limit value of physical blocks are pre-allocated to store the target file data, wherein the expected value is less than the upper limit value; After the physical block of the expected value is filled with the target file data, if an IO request for writing the target file data is received, the expected value is increased by a preset exponential level. The target file data is written based on the increased expected value of the physical blocks. Repeat the steps from receiving an IO request for a write operation on the target file data to writing the target file data to a physical block based on the increased expected value, until the increased expected value meets the preset condition of the upper limit value or no IO request for a write operation on the target file data is received.

8. An I / O performance optimization device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the method as described in any one of claims 1-7.

9. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform the steps of the method as described in any one of claims 1-7.

Images