Performance optimization method and device for first write-in after snapshot and medium

Abstract

The invention discloses a performance optimization method and device for first-time writing after snapshot and a medium, relates to the technical field of data writing, and provides the performance optimization method for first-time writing after snapshot aiming at the problem that after a Reflink technology is introduced into an OCFS2 file system, the performance is greatly reduced during first-time writing after snapshot creation. Direct IO writing is modified into buffer IO writing, and then data is brushed down to a disk through a dirty page brushing-down mechanism to complete data disk dropping. Based on the write modification operation, the write overhead can be reduced by using the high-speed characteristic of the memory, so that the effect of improving the write performance of the file system is achieved. Furthermore, according to the method, delayed downbrushing is adopted during downbrushing of the dirty pages, the time for writing the dirty pages back to the disk can be delayed on the premise that it is guaranteed that the data can finally fall into the disk in time, and therefore the situation that precious user data IO bandwidth is occupied by downbrushing of the dirty pages is avoided, and the effect of improving the writing performance of a file system is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data writing, in particular to a performance optimization method and device for first writing after snapshot and a medium. BACKGROUND

[0002] With the rapid development of information technology, file systems play a very key role in the field of data storage and management. OCFS2, as a high-performance cluster file system, is widely used in multi-node shared storage scenarios due to its excellent performance. In some current application scenarios, OCFS2 introduces Reflink technology to achieve data sharing and fast replication in the file system.

[0003] However, the Reflink technology has problems such as write amplification, which affects the overall performance of the OCFS2 file system. In particular, the first write after creating a snapshot will cause a significant decline in write performance. This problem seriously restricts the application of the OCFS2 file system in high-performance storage scenarios, affects the efficiency and experience of data writing for users, and urgently needs an effective solution. SUMMARY

[0004] The purpose of the present application is to provide a performance optimization method and device for first writing after snapshot and a medium, which solves the problem that the Reflink technology significantly affects the write performance of the file system when performing the first write after creating a snapshot.

[0005] To solve the above technical problems, the present application provides a performance optimization method for first writing after snapshot, comprising:

[0006] When performing the first write operation after creating a snapshot through Reflink, redirecting the first write operation to an in-memory buffer;

[0007] According to the system running state data of the file system, determining a dirty page flushing parameter combination;

[0008] According to the dirty page flushing parameter combination, delaying the dirty page in the in-memory buffer to which the first write operation is written to the disk to flush.

[0009] In an optional embodiment, delaying the dirty page in the in-memory buffer to which the first write operation is written to the disk to flush comprises:

[0010] Reorganizing the random dirty pages in the in-memory buffer according to physical continuity to obtain a dirty page combination;

[0011] For all dirty pages in each dirty page combination, aggregating and writing each dirty page to the disk.

[0012] In an alternative embodiment, after the first write operation is written to the dirty page in the memory buffer and is delayed to be flushed to the disk, further comprising:

[0013] Periodically scanning the fragmentation information of the files in the disk, and calculating the fragmentation degree value of the files according to the fragmentation information;

[0014] When the fragmentation degree value is greater than a preset fragmentation threshold, periodically acquiring the input / output load information of the file system;

[0015] When the input / output load information meets the low load condition, reorganizing the data blocks belonging to the same file in the disk to store the data blocks belonging to the same file continuously in the disk.

[0016] In an alternative embodiment, the disk is a virtual disk used by a virtual machine or a cloud platform;

[0017] The reorganization of the data blocks belonging to the same file in the disk to store the data blocks belonging to the same file continuously in the disk includes:

[0018] Through the online migration storage function of the virtual disk, the storage addresses of the data blocks of each file are re-allocated to realize the integration of the data blocks.

[0019] In an alternative embodiment, the redirection of the write operation to the memory buffer when the first write operation is created after the snapshot through the data block sharing technology includes:

[0020] Through the judgment logic added to the code path of the data block sharing write operation, it is identified whether the current data block sharing write operation is a first write operation;

[0021] If so, a memory buffer matching the size of the current write data is allocated through the kernel memory management interface;

[0022] Through the zero-copy data transmission mechanism, the to-be-written data in the user space is migrated to the allocated memory buffer.

[0023] In an alternative embodiment, the determination of the dirty page flushing parameter combination according to the system running state data of the file system includes:

[0024] The data importance level of the dirty page in the memory buffer and the fragmentation degree value of the disk are obtained; wherein, the data importance level is determined according to the file attributes of the corresponding file; the file attributes include: file type, storage path, user or application to which it belongs; the fragmentation degree value is determined according to the number of non-continuous data blocks in the disk and the total number of data blocks;

[0025] According to the system running state data, the data importance level and the fragmentation degree value, determine the dirty page flushing parameter combination.

[0026] In an alternative embodiment, according to the system running state data, the data importance level and the fragmentation degree value, determining the dirty page flushing parameter combination comprises:

[0027] Inputting the system running state data, the data importance level and the fragmentation degree value into a pre-trained decision model to obtain the dirty page flushing parameter combination output by the decision model;

[0028] The training step of the decision model comprises:

[0029] Obtain historical data of the file system; wherein the historical data comprises: system running state data, data importance level, fragmentation degree value, dirty page flushing parameter combination and system performance information;

[0030] According to the data consistency check result, mark each group of the historical data with a corresponding safety mark; wherein the data safety mark comprises a safe mark and an unsafe mark; when the file system is in the historical system running state data, and the dirty page in the memory buffer is flushed to the disk by the corresponding historical dirty page flushing parameter combination, if the flushed dirty page passes the data consistency check, mark the current group of the historical data with the safe mark, otherwise mark it with the unsafe mark;

[0031] Define a reward function with the system performance information and the safety mark as reward factors; wherein when the safety mark is the safe mark, and the system performance information represents the performance improvement of the file system, give a positive reward, otherwise give a negative reward;

[0032] Based on the historical data and the reward function, train a reinforcement learning model to obtain the decision model.

[0033] To solve the above technical problems, the application further provides a performance optimization device for first write after snapshot, comprising:

[0034] A write modification module is configured to redirect the first write operation to a memory buffer when the first write operation is performed after creating a snapshot through data block sharing;

[0035] A parameter determination module is configured to determine a dirty page flushing parameter combination according to system running state data of a file system;

[0036] Delaying dirty page flushing module, for writing the first write operation to the dirty page in the memory buffer according to the dirty page flushing parameter combination to delay the dirty page to the disk.

[0037] To solve the above technical problems, the application further provides a performance optimization device for first write after snapshot, comprising:

[0038] Memory, for storing computer programs;

[0039] Processor, for executing the computer programs to realize the steps of the performance optimization method for first write after snapshot.

[0040] To solve the above technical problems, the application further provides a non-volatile storage medium, wherein the non-volatile storage medium stores computer programs, and the computer programs are executed by a processor to realize the steps of the performance optimization method for first write after snapshot.

[0041] The application provides a performance optimization method for first write after snapshot, which redirects the first write operation after creating a snapshot of Reflink to a memory buffer, modifies direct IO writing to buffer IO writing, and stores data in the memory buffer as dirty pages, and then flushes the data to the disk through a dirty page flushing mechanism to complete data landing on the disk. Based on the above writing modification operation, the high-speed characteristics of the memory can be used to reduce the writing overhead, thereby effectively reducing the performance loss of the first write and improving the file system writing performance. Further, the method adopts delayed flushing when flushing dirty pages, which can delay the time of writing dirty pages to the disk on the premise of ensuring that data can be landed on the disk in time, thereby avoiding the occupation of valuable user data IO bandwidth by the dirty page flushing process (equivalent to the process of directly writing data to the disk), ensuring the efficiency of application IO, and achieving the effect of improving the file system writing performance. Therefore, the method improves the performance of first write after creating a snapshot of Reflink from the aspects of writing conversion and dirty page delayed flushing, thereby solving the problem that Reflink affects the OCFS2 file system writing performance.

[0042] The performance optimization device for first write after snapshot and the non-volatile storage medium provided by the application correspond to the above method and have the same effect. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort based on these drawings.

[0044] Figure 1 The flow chart of the performance optimization method of the first write after snapshot provided by the embodiment of the present application;

[0045] Figure 2 The flow chart of another performance optimization method of the first write after snapshot provided by the embodiment of the present application;

[0046] Figure 3 The structural diagram of the performance optimization method device of the first write after snapshot provided by the embodiment of the present application;

[0047] Figure 4 The structural diagram of another performance optimization method device of the first write after snapshot provided by the embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort are within the protection scope of the present application.

[0049] The core of the present application is to provide a performance optimization method, device and medium of the first write after snapshot.

[0050] In order to make the person in the art better understand the present application scheme, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0051] In the related art, Reflink adopts a read-on-write (RoW) mechanism, so that a certain degree of data sharing and fast replication can be achieved, but file system fragmentation is inevitably intensified. Fragmented data storage causes frequent disk seeking, which in turn greatly increases the disk seek time and reduces the disk input / output (IO) performance, and the read / write efficiency of the file is seriously affected. On the other hand, Reflink also has the problem of write amplification. A large amount of data writing will occupy the IO bandwidth of user data, thereby causing the user data writing performance to be damaged. In view of the above two aspects, although the OCFS2 (a high-performance cluster file system) file system introduces the Reflink technology to quickly create a data copy, the performance of the file system is affected. In particular, after creating a Reflink snapshot, the first write performance will be severely reduced, which needs to be optimized.

[0052] To solve the above problems, the present application provides a performance optimization method for the first write after snapshot, as shown in Figure 1 , which comprises:

[0053] S11: In the first write operation after creating a snapshot by data block sharing, the first write operation is redirected to the memory buffer.

[0054] S12: According to the system running state data of the file system, determine the dirty page flushing parameter combination.

[0055] S13: According to the dirty page flushing parameter combination, write the first write operation to the dirty page in the memory buffer to delay flushing to the disk.

[0056] For step S11, the first write after Reflink snapshot is modified from direct IO write to buffer IO write. The data originally written directly to the disk is first written to the memory buffer as a dirty page, and then the dirty page in the memory is flushed to the disk through the delay flushing mechanism of step S13 to complete the data landing.

[0057] As can be easily understood, based on the above step S11, the first write after Reflink snapshot does not need to write data directly to the disk, but is temporarily stored in the memory with high-speed characteristics. This has two advantages: first, it reduces the overhead of the first write operation by taking advantage of the high-speed characteristics of the memory, that is, it reduces the loss of system write performance; second, there is no need to immediately write data to the disk, so even if the current user is performing data writing, it will not squeeze the valuable user data IO bandwidth, ensuring the efficiency of application IO and improving the write performance of the file system.

[0058] For steps S12 and S13, even if the delay-flush mechanism is provided to enable the write data to achieve data flushing eventually. The focus of the delay-flush mechanism is "delay". That is, the dirty page flush in the memory buffer is not performed immediately after the first write operation, but the dirty page flush parameter combination (vm.dirty_xxx, which is used to determine various attributes including flush time) is determined through step S12, so as to avoid squeezing the user data IO bandwidth and reduce the impact on the system write performance.

[0059] It should be noted that the system running state data and the dirty page flush parameter combination are not strictly limited in the embodiment, and can be determined according to actual application. For example, the system running state data can include but is not limited to disk IO performance information, memory usage information, and data write information. The disk IO performance information can be obtained by reading the disk statistics information file ( / proc / diskstats file) in the file system. The / proc / diskstats file includes disk read / write request number, read / write sector number, sector size, disk rotation speed, and other statistics related to the disk, which can be used to represent the IO performance of the disk. Similarly, the memory usage information can be obtained by reading the memory information file ( / proc / meminfo file) in the file system. The / proc / meminfo file includes total memory (MemTotal), free memory (MemFree), memory used by buffer (Buffers), and memory used by cache (Cached). The data write information can be obtained by monitoring the write operation log of the file system directly, or indirectly by using the file system interface (such as inotify) to count the number of file writes and the amount of data written per unit time. In the Linux (an operating system kernel) system, the above monitoring can be realized by using the continuous monitoring file system write event command (inotifywait -m / path / to / filesystem -e write). The data write information mainly includes write frequency and write throughput. However, in most scenarios, the write throughput of the file system is very small, so only the write frequency can represent the data write situation of the file system. Further, the write frequency = write number / unit time, and the write throughput = write data amount / unit time.

[0060] In addition, the dirty page flushing parameter combination vm.dirty_xxx includes, but is not limited to, the background dirty page ratio threshold (vm.dirty_background_ratio), the synchronous disk flushing dirty page ratio threshold (vm.dirty_ratio), the dirty page maximum survival time (vm.dirty_expire_centisecs), and the dirty page writeback check interval (vm.dirty_writeback_centisecs), and the like. It should be noted that the dirty page flushing mechanism is also applied in the original file system, but in the original dirty page flushing mechanism, the above-mentioned parameters related to dirty page flushing are a fixed value configured in advance, and cannot be dynamically changed, nor can the delayed flushing be achieved. In step S12 of the method, the corresponding dirty page flushing parameter combination is determined according to the system running state data of the file system, so as to achieve the purpose of delaying the flushing of dirty pages, so as to avoid squeezing the user IO data write bandwidth. Moreover, the above-mentioned delay is dynamically changed, and is determined according to the real-time running state of the file system, so as to flush the dirty pages when the IO load of the file system is low, and to reduce the influence of dirty page flushing on the write performance of the file system to the maximum extent.

[0061] For example, it is assumed that the original dirty page flushing parameter combination is configured as follows:

[0062] vm.dirty_background_ratio = 10; vm.dirty_ratio = 20; vm.dirty_expire_centisecs = 3000; vm.dirty_writeback_centisecs = 500.

[0063] At a certain moment, it is found that the I / O delay of the disk is high (such as more than 50 ms), the memory usage is high (such as more than 80%), and the data write frequency is high (the write data amount is more than 10 MB / s). At this time, the time of delaying the flushing of dirty pages is delayed, and then the above-mentioned dirty page flushing parameter combination is adjusted as follows:

[0064] vm.dirty_background_ratio = 5 (the original value is 10), vm.dirty_ratio = 15 (the original value is 20), vm.dirty_expire_centisecs = 1000 (the original value is 3000), and vm.dirty_writeback_centisecs = 200 (the original value is 500).

[0065] Thus, the purpose of delaying dirty page flushing is achieved, the IO load peak of the file system is avoided, and the influence of dirty page flushing on the file system write performance is reduced. It should be noted that the above example is only one possible example. Since the definition and function of each dirty page flushing parameter are clear, when the need to extend or shorten the flushing delay time is determined according to different system IO loads, the specific values of each dirty page flushing parameter can be known, and the embodiment will not be described again.

[0066] As can be seen from the above, the performance optimization method for first write after snapshot provided by the application modifies the write mode of the first write operation of Reflink after creating a snapshot from direct IO write to buffer IO write, so that the data is first written into the memory buffer as a dirty page, and the subsequent data landing can be completed by using the existing dirty page flushing mechanism. This can reduce the write overhead by using the high-speed characteristics of the memory, thereby effectively reducing the performance loss of the first write and improving the file system write performance. In the subsequent process of flushing the dirty page to the disk, the IO load of the current file system can be determined by the system running state data, so as to dynamically adjust the dirty page flushing parameter combination, so that the dirty page flushing can be delayed to avoid the load peak of the file system, and the dirty page flushing can be performed during the period of low file system load, thereby avoiding the occupation of user data IO bandwidth and further reducing the influence of data write operation on the file system write performance.

[0067] As can be seen from the above, the method does not directly solve the write amplification problem of Reflink, but improves the write performance of the file system from two different angles of write conversion and dirty page delay flushing, thereby solving the problem that Reflink affects the write performance of the OCFS2 file system.

[0068] On the other hand, as can be seen from the above description, in addition to write amplification, the reason why Reflink affects the write performance of the OCFS2 file system also includes that the data fragmentation degree is serious. The embodiment provides an optional implementation scheme for solving the problem of serious data fragmentation, and the step S13 specifically comprises:

[0069] S131: recombining the random dirty pages in the memory buffer according to the physical continuity to obtain a dirty page combination.

[0070] S132: for all dirty pages in each dirty page combination, writing each dirty page to the disk.

[0071] That is, when the dirty pages in the memory buffer are flushed to the disk, the random dirty pages are reorganized according to the physical continuity (such as the logical block address) and written to the disk by the write-time data rearrangement technology, so as to reduce the degree of data fragmentation in the disk. Due to the temporary storage of the data in the memory buffer in step S11 of the method, the data blocks in the cache can be reorganized and written to the continuous address in the disk, so as to reduce the degree of data fragmentation in the disk.

[0072] For example, it is assumed that the data written to the memory buffer in a first data writing operation includes data block A, data block D and data block B. If the original Reflink direct IO writing method is used, the data is written to the disk in the order of data block A, data block D and data block B. However, in the present embodiment, the three data blocks are reordered according to the physical continuity as [data block A, data block B, data block D] based on the logical block address (LAB), and are written to the disk continuously, so as to achieve the effect of reorganization and aggregation, and the degree of data fragmentation in the disk can be reduced obviously.

[0073] Further, when the above-described de-fragmentation operation is implemented, the atomic transaction synchronous update of the extent tree and the copy-on-write (COW) reference mapping of the OCFS2 file system can be used. The update process is as follows: the old, fragmented extent is deleted; the new, continuous extent is inserted; and the latest COW reference mapping is synchronized and atomically committed to the system, so as to complete an update.

[0074] On the other hand, in addition to the de-fragmentation scheme provided by the above-described embodiment when the dirty pages are flushed to the disk, the present embodiment also provides an embodiment of actively de-fragmenting the data stored in the disk. The above-described method further includes the following steps after step S13:

[0075] S141: periodically scan the fragmentation information of the file in the disk, and calculate the fragmentation degree value of the file according to the fragmentation information.

[0076] S142: when the fragmentation degree value is greater than a preset fragmentation threshold, periodically acquire the input / output load information of the file system.

[0077] S143: when the input / output load information satisfies the low load condition, reorganize the data blocks belonging to the same file in the disk to store the data blocks belonging to the same file continuously in the disk.

[0078] It should be noted that the difference between the present embodiment and the above-described embodiment is that:

[0079] The defragmentation of the above embodiments is directed to data in the memory buffer (dirty pages). At this time, the data is in the memory, and does not need to be read from the disk, and will not excessively affect the IO load of the system. Moreover, since the data is written based on the Reflink using the RoW mechanism, there is a write amplification problem, and if it is directly flushed to the disk without processing, it will inevitably exacerbate the degree of data fragmentation in the disk. Therefore, the memory buffer defragmentation scheme in the above embodiments is necessary, and is executed in the process of each dirty page flushing.

[0080] In the present embodiment, the defragmentation is directed to data that has been written to the disk. On the one hand, the data needs to be read from the disk when reorganizing and aggregating the data, which will occupy a part of the IO resources. On the other hand, after reorganizing and aggregating, the data in the disk also needs to be migrated according to the aggregation result, which will also occupy a part of the IO resources. Therefore, when executing the disk data defragmentation scheme of the present embodiment, certain conditions need to be met, otherwise, the defragmentation effect may not be as expected (such as the degree of fragmentation in the disk is already very low), or even because of excessive occupation of system IO resources, the write performance of the file system will be affected, which is contrary to the original intention of defragmentation. Therefore, the present embodiment first judges through steps S141 and S142 before performing the defragmentation operation of step S143. Only when the degree of fragmentation in the disk reaches a certain degree and the current IO load of the file system is low, the defragmentation operation for the data in the disk is performed, so as to ensure that the write performance of the file system can be improved.

[0081] In addition, the fragmentation information in step S141 above, that is, various information that can reflect the degree of data fragmentation in the disk. The fragmentation information can include but is not limited to: the number of fragments, the size of the fragments, the position of the fragments in the disk, and the like. Based on the above fragmentation information, a specific value (i.e., the fragmentation degree value) can be calculated to quantify the degree of data fragmentation in the disk. In one possible embodiment, the fragmentation degree value = non-continuous block number / total block number*100%.

[0082] As for step S142 above, the IO load information also belongs to one of the system running state information. Specifically, the current IO load situation of the file system can be judged through the disk utilization rate, network transmission bandwidth and the like, which will not be described in detail in the present embodiment. However, it should be noted that the fragmentation degree value determines whether to trigger the defragmentation of the disk data, and the IO load information determines when to execute the defragmentation of the disk data.

[0083] Finally, how the above step S143 realizes the de-fragmentation of the data in the disk. One optional implementation is to use the existing OCFS2 file system's extended area end migration control interface (OCFS2_IOC_MOVE_EXT) ioctl, together with the user state tool defrag.ocfs2, to realize the online de-fragmentation operation of the OCFS2 file system. The user state tool defrag.ocfs2 reorganizes the scattered data segments of the file according to the scanning obtained fragment information, and stores the data blocks of each file continuously on the disk as much as possible, reduces the number of head movements, and improves the disk read-write efficiency.

[0084] On the other hand, the embodiment also provides another implementation scheme of the disk data de-fragmentation operation for the cloud platform scenario:

[0085] For the virtual disk used by the virtual machine or the cloud platform, the above step S143 is specifically: through the online migration storage function of the virtual disk, the storage address of each file data block is re-allocated to realize the data block integration.

[0086] For the online migration storage function provided in the virtual machine or the cloud platform, when realizing the migration of the data in the virtual disk, the re-allocation and the fragment integration of the disk file can be automatically realized, so as to reduce the degree of the disk data fragmentation, and no additional tool or script program is needed.

[0087] As known from the above, the embodiment provides an active de-fragmentation scheme for the disk data. When the degree of the data fragmentation in the disk is high, the degree of the fragmentation can be reduced by the embodiment, so as to reduce the number of head movements, improve the disk read-write efficiency, and achieve the purpose of improving the file system performance.

[0088] In addition, when the modification write, the dirty page delay flushing, and the de-fragmentation when flushing and the active de-fragmentation scheme provided in the above embodiment are implemented together, the flow of the method is as shown in Figure 2

[0089] ​Firstly, the step S11 is a write modification step, i.e. modifying the first write operation after the snapshot creation of the Reflink from direct IO write to buffer IO write. Then, the step S12 is a dirty page flushing parameter combination determination step, i.e. determining the dirty page flushing parameter combination for the dirty page delayed flushing according to the system running state information. After that, the step S13 is a dirty page delayed flushing step. However, it should be noted that the synchronous defragmentation operation during flushing is also needed when the dirty page delayed flushing is performed according to the steps S131-S132 of the above embodiment. Finally, the active defragmentation operation corresponding to the steps S141-S143 of the above embodiment is performed to reduce the defragmentation degree of the data stored in the disk.

[0090] On the other hand, the step S11 of the above embodiment requires that the first write operation after the snapshot creation of the Reflink is modified from direct IO write to buffer IO write. However, the above embodiment does not limit how to determine whether an operation or a write operation is the first write operation after the snapshot creation of the Reflink. It can be realized by a script file, a program, a code or an interrupt to determine whether the current operation is the first write operation after the snapshot creation of the Reflink when each operation or write operation is performed. It can also be realized by directly determining the code related to the first write operation after the snapshot creation of the Reflink and adding the judgment logic to determine whether it is the first write operation. The embodiment does not limit this. In addition, the above embodiment does not limit how to redirect the first write operation to the memory buffer.

[0091] Based on this, the embodiment provides an optional implementation scheme for the specific implementation of the step S11, and the step S11 specifically includes:

[0092] S111: identifying whether the current data block shared write operation is the first write operation by adding judgment logic to the code path of the data block shared write operation.

[0093] S112: if yes, allocating a memory buffer matching the size of the write data by the kernel memory management interface.

[0094] S113: migrating the to-be-written data in the user space to the allocated memory buffer by the zero-copy data transmission mechanism.

[0095] For the above step S111, the code involved in the Reflink write operation in the OCFS2 file system includes the write page function and direct IO function of the file system, etc. The code for checking the file inode state can be added to the code of these functions to realize the identification of the Reflink first write operation.

[0096] For the above step S112, when the Reflink first write operation is identified, the write operation needs to be redirected to the memory buffer, and the memory buffer needs to be pre-allocated, that is, corresponding to step S112 in the embodiment. When the memory buffer is allocated, because the memory resource is limited, a memory buffer of a size corresponding to the size of the write data of the current write operation needs to be allocated. The memory allocation operation can be realized by calling the memory management interface kmalloc() in the file system kernel. The size of the allocated memory buffer is usually an integer multiple of the file system page size (such as 4KB / 16KB), and the memory region is selected according to the system memory state when the memory buffer is allocated.

[0097] For the above step S113, after the memory buffer is allocated, the data is written into the corresponding memory buffer. In the embodiment, the zero-copy data transmission mechanism (such as the copy_from_iter() function from the iterator) provided by the file system kernel can be used to directly migrate the to-be-written data in the user space to the pre-allocated kernel buffer. This process avoids the double-copy overhead of the traditional data copy.

[0098] Further, as known from the above embodiment, after the write data is written into the memory buffer, the data is written to the disk by the original dirty page flushing mechanism. That is, the write data written into the memory buffer should be stored as dirty page data. For this, the embodiment provides an optional implementation scheme, after the above step S113, the method further includes:

[0099] S114: mapping the virtual address of the memory buffer to a physical page by using an address conversion function (virt_to_page()), and calling a file system core interface (add_to_page_cache_lru()) to formally register the physical page as a page cache member of an address space file (address_space).

[0100] S115: calling a set page dirty state function (set_page_dirty()) to mark the newly registered cache page as a dirty page.

[0101] S116: calling a page write function (writepage()) to write the data in the memory buffer to the disk.From then on, the written data can be registered as dirty pages in the memory buffer to trigger the delayed write-back mechanism of the Linux kernel. Subsequently, the dirty page flushing parameters are combined by the virtual machine (VM) subsystem to complete the data landing asynchronously under the premise of ensuring data security.

[0102] On the other hand, in the above step S12, a scheme for determining the dirty page flushing parameter combination by system running state data is given, that is, the dirty page delayed flushing realized thereby can avoid the load peak period of the file system, and the flushing of dirty pages is performed during the period when the load of the file system is low, and the data landing is completed, thereby avoiding the occupation of IO resources. However, the present application does not limit that the dirty page flushing parameter combination can only be determined by system running state data, and in actual application, other parameters can also be introduced according to different needs to determine the dirty page flushing parameter combination together.

[0103] For example, the present embodiment provides an optional implementation, and the above step S12 is specifically:

[0104] S121: Obtain the data importance level and fragmentation degree value of the dirty pages in the memory buffer.

[0105] Among them, the data importance level is determined according to the file attributes of the corresponding file. The file attributes include: file type, storage path, user or application to which it belongs. It should be noted that the present embodiment does not limit how many levels the data importance level has. For example, assuming that the data importance level has ten levels from 1 to 10, and the importance degree represented is increasing in turn. Further, an example of determining the data importance level is provided:

[0106] For the file with the storage directory of “ / var / lib / mysql / *”, the file type is a database file, which is extremely important, and can be assigned a data importance level of 10; for the file with the storage directory of “ / tmp / *”, the file type is a temporary file, which is not important, and can be assigned a data importance level of 1; for the file with the file extension of “.conf”, the file type is a configuration file, which is relatively important, and can be assigned a data importance level of 8.

[0107] In addition, the fragmentation degree value is a quantitative value used to represent the fragmentation degree of data in the disk in the above embodiment. It is determined according to the number of non-continuous data blocks in the disk and the total number of data blocks, specifically as in the above embodiment, fragmentation degree value = non-continuous block number / total block number*100%.

[0108] S122: Determine the dirty page flushing parameter combination according to the system running state data, the data importance level and the fragmentation degree value.

[0109] For step S122, it is not difficult to understand that the way of determining the dirty page flushing parameter combination based on the data importance level and the disk fragmentation index is the same as the principle of determining based on the system running state data in the above embodiment. For the data importance level, if the data importance level is high, the time of delaying flushing can be advanced to complete the landing of this important data as soon as possible, which is beneficial to guarantee data consistency. For the fragmentation degree value, if the value is large, it means that the fragmentation degree of the current disk is large, and the time of delaying flushing can be delayed to reduce the fragmentation degree of the disk through the active defragmentation operation provided in the above embodiment before new dirty page flushing (because new dirty page flushing generally causes the fragmentation degree of the disk to increase).

[0110] As can be seen from the above, the present embodiment takes the requirement of data importance in actual application and the disk fragmentation degree value into the consideration range of determining the dirty page flushing parameter combination. Thus, when delaying dirty page flushing, the delay time can meet the needs of different aspects in actual application, the time of delaying flushing is more appropriate, and the overall performance of the file system is further improved.

[0111] On the other hand, for how to determine the dirty page flushing parameter combination in step S12, a more direct solution is to directly determine the values of each parameter in the dirty page flushing parameter combination by manual operation. However, this solution depends on manual operation and is not easy to implement. In addition, it can also be obtained by increasing or adjusting the initial values of the original parameters. In an example given in the above embodiment, it has been given how to adjust the specific values of each dirty page flushing parameter when the load of the file system is large, so the present embodiment will not be repeated here. However, this adjustment scheme is still not accurate enough and will affect the effect of improving the write performance of the file system by the present method.

[0112] Moreover, when the data importance level and the disk fragmentation degree value are introduced as in the above embodiment, the dynamic adjustment and determination of the dirty page flushing parameter combination will become more complex and it is difficult to achieve by manual operation or simple increase or decrease adjustment. Based on this, the present embodiment provides an optional dirty page flushing parameter combination determination scheme, and step S12 is specifically:

[0113] Inputting the system running state data, the data importance level and the fragmentation degree value into a pre-trained decision model to obtain the dirty page flushing parameter combination output by the decision model.

[0114] The training step of the decision model includes:

[0115] S21: obtaining the historical data of the file system.

[0116] The historical data includes system running state data, data importance level, fragmentation degree value, dirty page flushing parameter combination, and system performance information.

[0117] S22: According to the data consistency verification result, mark each group of historical data with a corresponding security mark.

[0118] The data security mark includes a safe mark and an unsafe mark. When the file system is in the historical system running state data, and the dirty pages in the memory buffer are flushed to the disk by the corresponding historical dirty page flushing parameter combination, if the dirty pages are verified by the data consistency, the current group of historical data is marked as a safe mark, otherwise, it is marked as an unsafe mark.

[0119] S23: Defining a reward function with system performance information and security mark as reward factors.

[0120] When the security mark is a safe mark and the system performance information represents the performance improvement of the file system, a positive reward is given, otherwise, a negative reward is given. That is, in this embodiment, data consistency is taken as the highest priority reward factor. As long as the data consistency requirement is not met, a negative reward is given, so that the dirty page flushing parameter combination output by the decision model can meet the data consistency requirement when performing dirty page delayed flushing, so as not to cause data consistency to be not guaranteed because of flushing too late.

[0121] In addition, the system performance information described above can include but is not limited to system response time, throughput and other information. For the system response time parameter, the lower the parameter value, the better the performance of the file system, and a positive reward is given. For the throughput, the higher the parameter value, the better the performance of the file system, and a positive reward is given.

[0122] S24: Based on the historical data and the reward function, the reinforcement learning model is trained to obtain a decision model.

[0123] The reinforcement learning method used by the decision model in this embodiment is not limited, and an optional reinforcement learning algorithm is Q learning algorithm.

[0124] Therefore, the embodiment provides a dirty page flushing parameter combination determination scheme realized by using a reinforced machine learning model, which is suitable for a scenario in which multiple input parameters are used to determine a dirty page flushing parameter combination. The dirty page flushing parameter combination determined based on the embodiment is more accurate, and can maximize the improvement of the overall performance of the file system by the method. In addition, the decision model in the embodiment takes data consistency as the first priority for optimization, which maximizes the consistency of data when the dirty page flushing parameter combination output by the decision model is used for delayed flushing, and meets the data security requirements in actual applications.

[0125] In addition to the embodiments of the snapshot post-first write performance optimization method provided in the above embodiments, the present application also provides an embodiment corresponding to a computer program product. The computer program product comprises computer programs / instructions, which, when executed by a processor, can implement the steps of the snapshot post-first write performance optimization method according to any one of the above embodiments.

[0126] Since the embodiments of the computer program product part correspond to the embodiments of the method part, the embodiments of the computer program product part are described in the description of the embodiments of the method part, and will not be described here.

[0127] In the above embodiments, a snapshot post-first write performance optimization method is described in detail, and the present application also provides an embodiment corresponding to a snapshot post-first write performance optimization device. It should be noted that the embodiments of the device part are described from two angles, one based on functional modules and the other based on hardware.

[0128] Based on the functional module angle, the embodiment provides a snapshot post-first write performance optimization device, as shown in Figure 3 , which comprises:

[0129] The write modification module 11 is configured to redirect the first write operation to the memory buffer when the first write operation is performed after creating a snapshot through data block sharing.

[0130] The parameter determination module 12 is configured to determine a dirty page flushing parameter combination according to system running state data of the file system.

[0131] The delayed flushing module 13 is configured to flush the first write operation to the dirty page in the memory buffer to the disk according to the dirty page flushing parameter combination.

[0132] Since the embodiments of the device part correspond to the embodiments of the method part, the embodiments of the device part are described in the description of the embodiments of the method part, and will not be described here.

[0133] Figure 4 A structural diagram of a snapshot post-first write performance optimization apparatus provided by another embodiment of the present application is shown in FIG. 2. The snapshot post-first write performance optimization apparatus includes a memory 20 for storing a computer program. Figure 4

[0134] A processor 21 is configured to implement the steps of the snapshot post-first write performance optimization method according to any one of the above embodiments when executing the computer program.

[0135] The snapshot post-first write performance optimization apparatus provided by the embodiment can include, but is not limited to, a mobile terminal, a personal computer, a server, a workstation, etc.

[0136] The processor 21 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 21 can be implemented in at least one of a hardware form of a Digital Signal Processor (DSP), a Field-Programmable Gate Array (FPGA), and a Programmable Logic Array. The processor 21 can also include a main processor and a coprocessor. The main processor is a processor for processing data in a wake-up state, also known as a Central Processing Unit (CPU). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 21 can be integrated with a Graphics Processing Unit (GPU) for rendering and drawing the content to be displayed by the display screen. In some embodiments, the processor 21 can further include an Artificial Intelligence (AI) processor for processing machine learning-related computing operations.

[0137] ​The memory 20 can include one or more computer-readable storage media that can be non-transitory. The memory 20 can also include high-speed random access memory and nonvolatile, computer-readable storage media such as one or more magnetic disk storage devices, flash memory devices. In this embodiment, the memory 20 is at least used to store the following computer program 201, wherein the computer program is loaded and executed by the processor 21, and can realize the related steps of the snapshot post-first write performance optimization method disclosed in any of the foregoing embodiments. In addition, the resources stored by the memory 20 can also include an operating system 202 and data 203, etc., and the storage mode can be temporary storage or permanent storage. The operating system 202 can include Windows, Unix, Linux, etc. The data 203 can include but is not limited to a snapshot post-first write performance optimization method, etc.

[0138] In some embodiments, the snapshot post-first write performance optimization device can also include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0139] Those skilled in the art can understand that the structure shown in the above-mentioned embodiments does not constitute a limitation on the snapshot post-first write performance optimization device, and can include more or fewer components than those shown in the figure. Figure 4

[0140] The snapshot post-first write performance optimization device provided by the embodiment of the present application includes a memory and a processor, and the processor can realize the following method when executing the program stored in the memory: a snapshot post-first write performance optimization method.

[0141] Finally, the present application also provides an embodiment corresponding to a non-volatile storage medium. The non-volatile storage medium stores a computer program, and the computer program is executed by the processor to realize the steps recorded in the above-mentioned method embodiment.

[0142] It can be understood that if the method in the above-mentioned embodiments is realized in the form of a software function unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and executes all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes. ​

[0143] The above describes in detail the snapshot post-first write performance optimization method, device and medium provided by the present application. Each embodiment in the specification is described in a progressive manner, and each embodiment mainly explains the difference from other embodiments. The same or similar parts between each embodiment can be understood by referring to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant part can be understood by referring to the method part. It should be pointed out that for ordinary skilled in the art, without departing from the principle of the present application, the present application can be improved and modified, and these improvements and modifications also fall within the protection scope of the present application.

[0144] It should also be noted that in this specification, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

Claims

1. A method for performance optimization of first write after snapshot, characterized in that, The method comprises: redirecting the first write operation to a memory buffer when the first write operation is performed after creating a snapshot by data block sharing; determining a dirty page flushing parameter combination according to system running state data of a file system; writing the first write operation to the memory buffer for dirty page delayed flushing to a disk according to the dirty page flushing parameter combination.

2. The performance optimization method for snapshot first write of claim 1, wherein, The writing of the first write operation to the memory buffer for dirty page delayed flushing to a disk comprises: reorganizing random dirty pages in the memory buffer according to physical continuity to obtain dirty page combinations; for all dirty pages in each dirty page combination, aggregating and writing the dirty pages to the disk.

3. The performance optimization method for snapshot first write of claim 1, wherein, After the writing of the first write operation to the memory buffer for dirty page delayed flushing to a disk, the method further comprises: periodically scanning fragmentation information of files in the disk, and calculating a fragmentation degree value of the files according to the fragmentation information; when the fragmentation degree value is greater than a preset fragmentation threshold, periodically acquiring input / output load information of the file system; when the input / output load information satisfies a low load condition, reorganizing data blocks belonging to the same file in the disk to store the data blocks belonging to the same file continuously in the disk.

4. The performance optimization method for snapshot first write of claim 3, wherein, The disk is a virtual disk used by a virtual machine or a cloud platform. The reorganizing of the data blocks belonging to the same file in the disk to store the data blocks belonging to the same file continuously in the disk comprises: reassigning storage addresses of file data blocks to realize data block integration through an online migration storage function of the virtual disk.

5. The performance optimization method for write-once after snapshot of any of claims 1 to 4, wherein, The redirecting of the first write operation to the memory buffer when the first write operation is performed after creating a snapshot by data block sharing comprises: identifying whether the current data block shared write operation is a first write operation through judgment logic added to a code path of the data block shared write operation; if yes, allocating a memory buffer matching the size of the write data through a kernel memory management interface; migrating the to-be-written data in a user space to the allocated memory buffer through a zero-copy data transmission mechanism.

6. The performance optimization method for write-once after snapshot of any one of claims 1 to 4, wherein, The determining of the dirty page flushing parameter combination according to system running state data of a file system comprises: obtaining a data importance level of dirty pages in the memory buffer and a fragmentation degree value of the disk; wherein the data importance level is determined according to file attributes of corresponding files; the file attributes comprise: file type, storage path, user or application to which the file belongs; the fragmentation degree value is determined according to the number of non-continuous data blocks in the disk and the total number of data blocks; determining the dirty page flushing parameter combination according to the system running state data, the data importance level and the fragmentation degree value.

7. The performance optimization method for snapshot first write of claim 6, wherein, The determining of the dirty page flushing parameter combination according to the system running state data, the data importance level and the fragmentation degree value comprises: inputting the system running state data, the data importance level and the fragmentation degree value into a pre-trained decision model to obtain the dirty page flushing parameter combination output by the decision model; The training step of the decision model comprises: obtaining historical data of the file system; wherein the historical data comprises: system running state data, data importance level, fragmentation degree value, dirty page flushing parameter combination and system performance information; According to the data consistency check result, mark each group of the historical data with corresponding safety mark; wherein the data safety mark includes safety mark and unsafe mark; when the file system is in the historical system running state data, and the delayed flushing of the dirty page in the memory buffer is performed through the corresponding historical dirty page flushing parameter combination, if the flushed dirty page to the disk passes the data consistency check, mark the current group of the historical data with the safety mark, otherwise mark it with the unsafe mark; The system performance information and the safety mark are used as reward factors to define a reward function; wherein when the safety mark is the safety mark, and the system performance information represents the performance improvement of the file system, a positive reward is given, otherwise a negative reward is given; Based on the historical data and the reward function, the reinforcement learning model is trained to obtain the decision model.

8. A performance optimization device for the first write after a snapshot, characterized in that, Comprise: The write modification module is used for redirecting the first write operation to the memory buffer when the first write operation is performed after creating the snapshot through the data block sharing; The parameter determination module is used for determining the dirty page flushing parameter combination according to the system running state data of the file system; The delayed flushing module is used for flushing the first write operation to the dirty page in the memory buffer to the disk according to the dirty page flushing parameter combination.

9. A performance optimization device for the first write after a snapshot, characterized in that, Comprise: Memory, used for storing computer programs; The processor is used for executing the computer program to realize the steps of the performance optimization method of the first write operation after the snapshot as claimed in any one of claims 1 to 7.

10. A non-volatile storage medium, comprising: The non-volatile storage medium stores a computer program, and the computer program is executed by the processor to realize the steps of the performance optimization method of the first write operation after the snapshot as claimed in any one of claims 1 to 7.

Images