Write request processing method and device, computer equipment and storage medium

Abstract

The invention relates to a write request processing method and device, computer equipment and a storage medium. The method comprises the following steps: determining the type of a to-be-processed write request according to a preset classification threshold value and the data volume of the received to-be-processed write request; if the to-be-processed write request is a first type of write request, splitting data corresponding to the to-be-processed write request into a logic alignment part and a logic non-alignment part; the data volume of the first type of write request is greater than the classification threshold; and writing the logic alignment part into a data disk and / or processing the logic non-alignment part by adopting a log pre-writing mode. By adopting the method, the service life of the log disk can be prolonged.

Description

Technical Field

[0001] The present application relates to the field of distributed storage technology, and in particular to a write request processing method, apparatus, computer equipment and storage medium. Background Art

[0002] Distributed storage systems store data in multiple independent data servers. Distributed storage technology can improve system reliability and scalability to meet different usage scenarios. In order to give full play to the write performance of the disk, data is usually appended in the data persistence layer by appending writes, and an index layer is added on top of the persistence layer to record the location information of the data.

[0003] In traditional technology, write-ahead log technology can be used to support additional writes to disks. Through write-ahead log technology, I / O requests are pre-written to a dedicated log disk for persistence, and then each I / O request is saved in the cache. When enough data is accumulated in the cache, the data in the cache is flushed to the data disk.

[0004] However, the conventional technology has the problem that a large amount of storage space of the log disk is occupied, which affects the service life of the log disk. Summary of the invention

[0005] Based on this, it is necessary to provide a write request processing method, device, computer equipment and storage medium that can extend the service life of the log disk in response to the above technical problems.

[0006] In a first aspect, the present application provides a write request processing method, comprising:

[0007] Determining the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request;

[0008] If the pending write request is a write request of the first type, splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part; and the data volume of the first type of write request is greater than the classification threshold;

[0009] The logically aligned portion is written into a data disk and / or the logically non-aligned portion is processed by using a write-ahead log.

[0010] In one of the embodiments, splitting the data corresponding to the pending write request into a logically aligned portion and a logically non-aligned portion includes:

[0011] According to the cache data in the cache data block corresponding to the pending write request and a preset decomposition threshold, the data corresponding to the pending write request is split into a logically aligned part and a logically non-aligned part.

[0012] In one of the embodiments, splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part according to the cached data in the cached data block corresponding to the pending write request and a preset decomposition threshold comprises:

[0013] Merging the cached data and the data corresponding to the pending write request to obtain merged data;

[0014] The data in the merged data whose data volume is greater than the preset decomposition threshold is determined as the logically aligned part, and the data in the merged data whose data volume is not greater than the preset decomposition threshold is determined as the logically non-aligned part.

[0015] In one embodiment, the processing of the logical non-aligned portion by using a write-ahead log includes:

[0016] A log entry of the logically non-aligned portion is generated, and the log entry is written into a log disk, and the logically non-aligned portion is written into a new cache data block.

[0017] In one embodiment, the method further comprises:

[0018] If the pending write request is a write request of the second type, data corresponding to the pending write request is processed by using a pre-write log method; and the data volume of the second type of write request is not greater than the classification threshold.

[0019] In one embodiment, the method further comprises:

[0020] Upon receiving the pending write request, determining remaining space on the log disk;

[0021] The classification threshold is determined according to the remaining space of the log disk.

[0022] In a second aspect, the present application further provides a write request processing device, comprising:

[0023] A first determination module, configured to determine the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request;

[0024] a splitting module, configured to split data corresponding to the pending write request into a logically aligned part and a logically non-aligned part if the pending write request is a first type of write request; and the data volume of the first type of write request is greater than the classification threshold;

[0025] The first processing module is used to write the logically aligned part into a data disk and / or process the logically non-aligned part by using a write-ahead log.

[0026] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0027] Determining the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request;

[0028] If the pending write request is a write request of the first type, splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part; and the data volume of the first type of write request is greater than the classification threshold;

[0029] The logically aligned portion is written into a data disk and / or the logically non-aligned portion is processed by using a write-ahead log.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the following steps are implemented:

[0031] Determining the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request;

[0032] If the pending write request is a write request of the first type, splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part; and the data volume of the first type of write request is greater than the classification threshold;

[0033] The logically aligned portion is written into a data disk and / or the logically non-aligned portion is processed by using a write-ahead log.

[0034] In a fifth aspect, the present application further provides a computer program product, including a computer program, which implements the following steps when executed by a processor:

[0035] Determining the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request;

[0036] If the pending write request is a write request of the first type, splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part; and the data volume of the first type of write request is greater than the classification threshold;

[0037] The logically aligned portion is written into a data disk and / or the logically non-aligned portion is processed by using a write-ahead log.

[0038] The above-mentioned write request processing method, device, computer equipment and storage medium can determine the type of the pending write request according to the preset classification threshold and the data volume of the received pending write request, so that when the pending write request is a first type of write request, the data corresponding to the pending write request can be split into a logically aligned part and a logically non-aligned part, wherein the data volume of the first type of write request is greater than the classification threshold, and then the logically aligned part is written to the data disk and / or the logically non-aligned part is processed by pre-writing log. In this way, when the data volume of the received pending write request is large, by splitting the data corresponding to the processor write request, different processing methods are used for different split data, the logically aligned part of the data corresponding to the write request is written to the data disk, and the logically non-aligned part is processed by pre-writing log. This can greatly reduce the amount of data written to the log disk, reduce the double writing problem, save the service life of the log disk with limited performance loss, and release the log items of the write request with a small amount of data occupying the log disk in time, reclaim the space of the log disk in time, improve the throughput in high concurrency scenarios, and thus extend the service life of the log disk. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] FIG1 is an application environment diagram of a write request processing method in one embodiment;

[0041] FIG2 is a schematic diagram of a flow chart of a write request processing method in one embodiment;

[0042] FIG3 is a schematic flow chart of a write request processing method in another embodiment;

[0043] FIG4 is a flow chart of a write request processing method in another embodiment;

[0044] FIG5 is a schematic flow chart of a write request processing method in another embodiment;

[0045] FIG6 is a block diagram of a write request processing device in one embodiment;

[0046] FIG7 is a structural block diagram of a write request processing device in another embodiment;

[0047] FIG8 is a structural block diagram of a write request processing device in another embodiment;

[0048] FIG9 is a structural block diagram of a write request processing device in another embodiment;

[0049] FIG10 is a structural block diagram of a write request processing device in another embodiment;

[0050] FIG. 11 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] The write request processing method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 102 communicates with the computer device 104 through the network. The computer device 104 can receive the write request sent by the terminal 102 and process the write request sent by the terminal 102. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head-mounted device can be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc.

[0053] In an exemplary embodiment, as shown in FIG2 , a write request processing method is provided, which is described by taking the method applied to the computer device in FIG1 as an example, and includes the following steps:

[0054] S201, determining the type of the pending write request according to a preset classification threshold and the data volume of the received pending write request.

[0055] Among them, the type of write request to be processed can be a first type of write request, or it can be a second type of write request. The data volume of the first type of write request is greater than the preset classification threshold, and the data volume of the second type of write request is not greater than the preset classification threshold. That is to say, the first type of write request can be a large IO request, and the second type of write request can be a small IO request.

[0056] Optionally, the preset classification threshold in this embodiment may be an empirical value determined based on historical data, or may be a threshold determined in real time based on the remaining space of the storage space. Exemplarily, assuming that the preset classification threshold is 256K, if the amount of data received for the pending write request is 512K, then the type of the pending write request may be determined to be the first type, that is, the pending write request is a large IO request, and if the amount of data received for the pending write request is 102K, then the type of the pending write request may be determined to be the second type.

[0057] S202: If the pending write request is a first type of write request, split data corresponding to the pending write request into a logically aligned part and a logically non-aligned part; the data volume of the first type of write request is greater than a classification threshold.

[0058] In this embodiment, if it is determined that the pending write request is the above-mentioned first type of write request, the data corresponding to the pending write request can be split, and the data corresponding to the pending write request can be split into a logically aligned part and a logically non-aligned part. Optionally, in this embodiment, the splitting threshold can be determined according to the triggering threshold for data flushing in the cache data block. For example, the triggering threshold for triggering data flushing can be determined as the splitting threshold, or the adjusted value of the triggering threshold can be determined as the splitting threshold. Optionally, in this embodiment, the data corresponding to the splitting threshold in the data corresponding to the pending write request can be determined as the logically aligned part, and the remaining data can be determined as the logically non-aligned part.

[0059] S203, writing the logically aligned part into the data disk and / or processing the logically non-aligned part by using a write-ahead log.

[0060] Among them, the write-ahead log technology is usually used to support disk append writing. The write-ahead log writes I / O requests to a dedicated log disk for persistence, and then saves each I / O request in the cache. When the data accumulated in the cache is sufficient, it will be flushed to the data disk. On the one hand, the write-ahead log can organize the data to be flushed according to the data organization method of append writing by caching the data of multiple write requests in the cache. The data in the cache can be directly written to the disk during the flushing process, which meets the append writing feature without generating holes. On the other hand, the write-ahead log can merge I / O requests and reduce the number of writes to the data disk, which is particularly effective for small I / O requests. However, the write-ahead log will cause double writing of data, that is, the data needs to be written to the log disk once and to the data disk once. The more times it is written, the greater the loss of disk life. However, for small I / O requests, the frequency of writing to the data disk is much lower than the frequency of writing to the log disk. In the final analysis, the double write situation is not serious. However, for large I / O requests, after being protected by replicas or erasure codes, they may still be large I / O requests before persistence. Large I / O requests will quickly fill up the cache and trigger the flush process. After writing to the log disk, they will soon be written to the data disk through the flush process. The larger the I / O request, the more serious the double write.

[0061] Therefore, in this embodiment, when the pending write request is a write request of the first type, that is, the pending write request is a large IO request, the data corresponding to the received large IO request is split into a logically aligned part and a logically non-aligned part, and the logically aligned part is written to the data disk, and the logically non-aligned part is processed by a preset log; when the pending write request is a write request of the second type, that is, the pending write request is a small IO request, the data corresponding to the pending request is processed by a preset log, so that the disk double writing problem caused by the pre-write log is greatly optimized under the condition of very limited impact on performance, and the service life of high-speed storage media can be greatly saved, and the append writing feature of the data disk can be guaranteed, and the space of the log disk can be recovered in time, and the throughput in high concurrency scenarios can be improved. In addition, if large I / O requests are continuously received, the above process is repeated. In fact, the amount of data written to the log disk is only the logically non-aligned part, which can greatly reduce the consumption of the log disk by the write request and ensure the append writing feature of the data disk. Since it is an append write, the performance of the disk will be relatively high. Compared with writing all data to the log disk, the append write can make the performance difference caused by large I / O request data directly falling to the data disk small enough, greatly reducing double writing with small performance sacrifice, thus saving the service life of the log disk.

[0062] In the above-mentioned write request processing method, since the type of the pending write request can be determined according to the preset classification threshold and the data volume of the received pending write request, when the pending write request is a first type of write request, the data corresponding to the pending write request can be split into a logically aligned part and a logically non-aligned part, wherein the data volume of the first type of write request is greater than the classification threshold, and then the logically aligned part is written into the data disk and / or the logically non-aligned part is processed by a pre-write log method. In this way, when the data volume of the received pending write request is large, by splitting the data corresponding to the processor write request, different processing methods are used for different split data, the logically aligned part of the data corresponding to the write request is written into the data disk, and the logically non-aligned part is processed by a pre-write log method, which can greatly reduce the amount of data written to the log disk, reduce the double writing problem, save the service life of the log disk with limited performance loss, and release the log items of the write request with a small amount of data occupying the log disk in time, reclaim the space of the log disk in time, improve the throughput in high concurrency scenarios, and thus extend the service life of the log disk.

[0063] In some scenarios, the cache data block corresponding to the pending write request may contain cached data that has already been cached. In an exemplary embodiment, the above S202 includes:

[0064] Step A: splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part according to the cache data in the cache data block corresponding to the pending write request and a preset splitting threshold.

[0065] Optionally, in this embodiment, the cache data in the cache data block corresponding to the pending write request can be split according to a preset decomposition threshold, and the data corresponding to the pending write request can be split according to a preset decomposition threshold, and then the split cache data and the split data corresponding to the pending write request that can satisfy the triggering of the flushing process are merged together to form a logically aligned part, and the remaining other data is used as a logically non-aligned part. Optionally, the determination principle of the preset decomposition threshold in this embodiment can be referred to the description in the above embodiment, and this embodiment will not be repeated here.

[0066] In this embodiment, the process of splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part according to the cache data in the cache data block corresponding to the pending write request and a preset decomposition threshold is relatively simple, and the logically aligned part and the logically non-aligned part corresponding to the pending write request can be quickly obtained. In addition, splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part provides the possibility of using different processing methods for different split data, thereby greatly reducing the amount of data written to the log disk, reducing the double writing problem, and saving the service life of the log disk with limited performance loss. The log items of write requests with a smaller amount of data occupying the log disk will be released in time, and the space of the log disk will be reclaimed in time, thereby improving the throughput in high concurrency scenarios, thereby extending the service life of the log disk.

[0067] In this embodiment, the specific implementation of the above step A will be specifically described. In one embodiment, as shown in FIG3 , the above step A includes:

[0068] S301, merging cached data and data corresponding to a pending write request to obtain merged data.

[0069] Optionally, in this embodiment, the cached data in the above-mentioned cached data block and the data corresponding to the pending write request can be spliced, and the cached data and the data corresponding to the pending write request can be merged together to obtain the merged data of the cached data and the data corresponding to the pending write request. For example, the starting data of the data corresponding to the pending write request can be spliced ​​after the ending data of the cached data to obtain the merged data, or the starting data of the cached data can be spliced ​​after the ending data of the data corresponding to the pending write request to obtain the merged data.

[0070] S302, determining the data in the merged data whose data volume is greater than the preset decomposition threshold as the logically aligned part, and determining the data in the merged data whose data volume is not greater than the preset decomposition threshold as the logically non-aligned part.

[0071] Optionally, in this embodiment, the merged data can be divided according to a preset decomposition threshold, and the data in the merged data whose data volume is greater than the preset decomposition threshold is determined as a logically aligned part, and the data in the merged data whose data volume is less than or equal to the preset decomposition threshold is determined as a logically non-aligned part. For example, if the data volume represented by the preset decomposition threshold is 256K, the merged data can be divided according to 256K. For example, if the data volume of the merged data is 600K, the data in the merged data that meets 256K can be determined as the first logically aligned part starting from the starting position of the merged data, and the division can continue from the end position of the first logically aligned part, and the data that meets 256K starting from the end position of the first logically aligned part can be determined as the second logically aligned part, and the remaining data can be determined as the logically non-aligned part.

[0072] In this embodiment, by merging the cached data and the data corresponding to the pending write request, merged data can be obtained, so that the data in the merged data whose data volume is greater than the preset decomposition threshold can be determined as a logically aligned part, and the data in the merged data whose data volume is not greater than the preset decomposition threshold can be determined as a logically non-aligned part. In this way, in the process of processing the received pending write request, the cached data in the cache data block corresponding to the pending write request can also be taken into consideration, so that the cached data in the cache data block can be dropped to disk at the same time as the data corresponding to the pending write request is dropped to disk.

[0073] In this embodiment, the specific implementation of the preset log is explained. In one embodiment, the "using a pre-write log to process the logical non-aligned part" in S203 includes:

[0074] Step B: Generate a log entry for the logical non-aligned portion, write the log entry into a log disk, and write the logical non-aligned portion into a new cache data block.

[0075] In this embodiment, for the above-mentioned logical non-aligned part that does not support direct placement on the data disk, the logical non-aligned part can be cached in other cache data blocks first, and the log entry of the logical non-aligned part is generated and written to the log disk to complete persistence, which can ensure that data will not be lost in abnormal scenarios. When all data persistence is completed, that is, all data written to the data disk and log disk are successfully written, and the persistent index metadata is successful, the response to the write request is completed, and the log disk space and cache data block space corresponding to the data that has been flushed are released.

[0076] Usually, the log disk uses high-speed storage media, such as NVMe SSD, etc. The data disk supports various types of devices, for example, storage media such as NVMe SSD or SATA SSD. The log disk will also establish a separate partition to store chunk metadata. Chunk is a logical unit for storing data in the persistence layer. The metadata of the chunk is stored in the metadata partition, and the data is stored in the data disk. It supports reading and writing multiple chunks at the same time, and each physical disk will store data of multiple chunks. The data of the chunk is written in append, and the append write adopts a fixed organization method. Optionally, in this embodiment, the logical non-aligned part can be divided according to the 4K granularity, and each 4K data corresponds to 64B of metadata. The metadata includes index data on the persistence layer and the verification information and metadata information generated by the persistence layer itself. Based on the pre-write log technology, the data is organized in the append write format in the memory, and the data background is flushed to the data disk after the disk drop conditions are met, which not only ensures the append write of the disk, but also avoids the holes caused by alignment or 0 padding. That is to say, when data is organized in the memory in the append-write format, 4K of original data + 64B of index data, each minimum data unit length is 4160Bytes. In order to be able to append to the data disk without generating holes or filling 0s, the data length and starting address of each refresh need to be aligned with the data disk sector.

[0077] In this embodiment, for the logically non-aligned part of the write request to be processed, a log entry for the logically non-aligned part is generated, and the log entry is written to the log disk. The logically non-aligned part is written to a new cache data block, which can ensure that data will not be lost in abnormal scenarios, and can greatly reduce the amount of data written to the log disk, reduce the double write problem, and save the service life of the log disk with limited performance loss.

[0078] In the above scenario where the type of pending write request is determined based on the preset classification threshold and the amount of data of the pending write request received, the preset classification threshold can be determined in real time based on the remaining space of the log disk. In this embodiment, the process of determining the classification threshold is described in detail. In one embodiment, as shown in FIG4 , the above method further includes:

[0079] S401, when receiving a pending write request, determining the remaining space of the log disk.

[0080] In this embodiment, when a pending write request is received, the used space in the log disk may be determined, and the remaining space of the log disk may be determined based on the used space in the log disk.

[0081] S402: Determine a classification threshold according to the remaining space of the log disk.

[0082] Optionally, in this embodiment, the principle of determining the classification threshold according to the remaining space of the log disk can be: when the remaining space of the log disk is greater than or equal to 30%, the classification threshold can be set to 768K; when the remaining space is less than 30%, the classification threshold decreases in a gradient, for example, when the remaining space is 20%, the classification threshold can be 512K, and when the remaining space is 10%, the classification threshold can be 256K. It should be noted that the above data is only an example, and the specific setting of the classification threshold is not limited to the above data.

[0083] In this embodiment, when a pending write request is received, by determining the remaining space of the log disk, the classification threshold can be determined in real time according to the remaining space of the log disk, so that the determination of the classification threshold is more accurate, thereby making the type of the pending write request determined according to the classification threshold more accurate, thereby improving the accuracy of determining the type of the pending write request.

[0084] To facilitate understanding by those skilled in the art, as shown in FIG5 , the write request processing method provided by the present disclosure is described in detail below. The method may include:

[0085] As shown in Figure 5, the data corresponding to the cache data in cache data block 1 has been persisted to the log disk, corresponding to Seq1, Seq2, and Seq6 in Figure 5. Seq represents a log entry, which consists of log metadata information, valid data, and alignment padding data. Alignment padding ensures that it can be written to the log disk. The metadata information of the log describes the valid data interval of the log, which is used for reloading and restoring data.

[0086] When a new I / O to be processed is received, an I / O classification threshold is first determined based on the remaining space on the log disk, and then it is determined whether it is a large I / O or a small I / O based on the threshold. If it is a large I / O and there is cached data that has not been flushed in cache data block 1, the cached data and the valid data carried by the large I / O are first merged in order, and then the merged data is trimmed: divided into logically aligned parts and logically non-aligned parts. The logically aligned part supports being written to the data disk and is directly written to the data disk; the logically non-aligned part does not support being written directly to the disk and needs to be cached to cache data block 2, and a Seq7 is generated to write to the log disk to complete persistence, which can ensure that data will not be lost in abnormal scenarios. When all data is persisted, that is, all data written to the data disk and log disk are written successfully, and the persistent index metadata is successful, the response to the write request is completed, and the log disk space (Seq1, Seq2, Seq6) corresponding to the data that has been flushed and the space of cache data block 1 are released.

[0087] The following is an explanation of the write request processing method provided by this application using a specific example:

[0088] S1, deploy three disk management nodes in a three-copy manner, run a disk management process on each node, and the disk management processes of the three nodes are s0, s1, and s2 respectively.

[0089] S2, each disk management process manages 16 NVMe SSD disks in this node, of which 8 are TLC media and 8 are QLC media. s0 manages TLC disks t0-t7 and QLC disks q0-q7, s1 manages TLC disks t7-t15 and QLC disks q7-q15, and s2 manages TLC disks t16-t23 and QLC disks q16-q23. TLC disks are used as log disks and QLC disks are used as data disks.

[0090] S3 deploys three cluster management nodes m0, m1 and m2, where m0 is the primary node and m1 and m2 are standby nodes. s0-s2 regularly report disk status to m0, and m0 records the status of all disks in the cluster. Based on the three-copy data protection strategy, m0 selects three disks from the cluster to join the data protection group. For balance, one disk will be selected from each of s0-s2.

[0091] S4 deploys a persistence layer client process c0, which sends a write request to the distributed storage cluster through the remote procedure call protocol. The I / O size is 512K, and the request will be distributed to s0, s1 and s2.

[0092] S5, the disk management process s0 receives a write request and first determines the I / O classification threshold based on the remaining space on the current log disk. It determines that it is a large I / O based on the threshold, and the program enters the large I / O processing flow.

[0093] S6, obtain the associated cache data block handle and check whether there is cache data to be flushed in the data block. If so, the cached data is represented by b0. The cache data b0 and the large I / O are merged into a large buffer data area buffer1. The log entry corresponding to b0 in the log disk t0 is seq0. seq contains metadata information describing the valid data interval and data after alignment and padding with 0 (aligned according to the sector size of the log disk).

[0094] S7, split buffer1 into two parts according to the alignment unit (256K+4K: the minimum data unit is 4K+64B. In order to ensure that it can be written to the disk, 64 minimum data units, namely 256K+4K, need to be gathered. The general sector size of the data disk is 512B or 4K): seg1 represents the aligned data in the front and supports direct writing to the data disk. seg2 represents the unaligned data at the end (if any) and cannot be written to the disk directly.

[0095] S8, s0 submits an asynchronous write request to the data disk q0 and the log disk t0. The content written to q0 is seg1, and the log entry seq1 containing seg2 is written to t0. seg2 is written to the cache data block buffer2. At the same time, the metadata meta1 that records the location of valid data on the data disk is written to the metadata partition. The metadata partition is generally deployed together with the log disk, which is equivalent to a part of the log disk.

[0096] S9, after seg1, seg2, seq1 and meta1 are written successfully, s0 returns a write success message to c0;

[0097] S10, s0 releases the log entry seq0 that has been written to the data disk q0, and releases the cached data block buffer1.

[0098] In summary, by optimizing the large I / O processing flow, most of the data brought by the large I / O is written less to the log disk t0: if the effective data length of the large I / O is L1, the length of seg2 is L2, and if L1 is written to the log disk, the metadata and alignment length generated is L3, where L1 is much larger than L2, and the length written less to the log disk t0 is (L1-L2+L3).

[0099] It should be noted that for the description of the above steps, reference may be made to the relevant description in the above embodiment, and the effects are similar, so this embodiment will not be repeated here.

[0100] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0101] Based on the same inventive concept, the embodiment of the present application also provides a write request processing device for implementing the write request processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in one or more write request processing device embodiments provided below can refer to the limitations on the write request processing method above, and will not be repeated here.

[0102] In an exemplary embodiment, as shown in FIG6 , a write request processing device is provided, comprising: a first determining module 10, a splitting module 11 and a first processing module 12, wherein:

[0103] The first determination module 10 is used to determine the type of the pending write request according to a preset classification threshold and the data volume of the received pending write request.

[0104] The splitting module 11 is used to split the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part if the pending write request is a first type of write request; the data volume of the first type of write request is greater than the classification threshold.

[0105] The first processing module 12 is used to write the logically aligned part into the data disk and / or process the logically non-aligned part by using a write-ahead log.

[0106] The write request processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0107] Based on the above embodiment, as shown in FIG7 , optionally, the above splitting module 11 includes: a splitting unit 111, wherein:

[0108] The splitting unit 111 is used to split the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part according to the cache data in the cache data block corresponding to the pending write request and a preset splitting threshold.

[0109] The write request processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0110] Based on the above embodiment, please continue to refer to Figure 7. Optionally, the above splitting unit 111 is specifically used to merge the cached data and the data corresponding to the pending write request to obtain merged data; determine the data in the merged data whose data volume is greater than the preset decomposition threshold as a logically aligned part, and determine the data in the merged data whose data volume is not greater than the preset decomposition threshold as a logically non-aligned part.

[0111] The write request processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0112] Based on the above embodiment, as shown in FIG8 , optionally, the above first processing module 12 includes: a processing unit 121, wherein:

[0113] The processing unit 121 is used to generate a log entry of the logical non-aligned part, write the log entry into a log disk, and write the logical non-aligned part into a new cache data block.

[0114] The write request processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0115] Based on the above embodiment, as shown in FIG9 , optionally, the above device further includes: a second processing module 13, wherein:

[0116] The second processing module 13 is used to process the data corresponding to the write request to be processed by using a pre-write log method if the write request to be processed is a write request of the second type; the data volume of the write request of the second type is not greater than the classification threshold.

[0117] The write request processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0118] On the basis of the above embodiment, as shown in FIG10 , optionally, the above device further includes: a second determination module 14 and a third determination module 15, wherein:

[0119] The second determination module 14 is configured to determine the remaining space of the log disk when receiving a pending write request.

[0120] The third determination module 15 is used to determine the classification threshold according to the remaining space of the log disk.

[0121] The write request processing device provided in this embodiment can execute the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.

[0122] Each module in the above-mentioned write request processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module above.

[0123] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG11. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store cache data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a write request processing method is implemented.

[0124] Those skilled in the art will understand that the structure shown in FIG. 11 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0125] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0126] Determine the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request;

[0127] If the pending write request is a first type of write request, splitting data corresponding to the pending write request into a logically aligned part and a logically non-aligned part; and the data volume of the first type of write request is greater than the classification threshold;

[0128] The logically aligned part is written to the data disk and / or the logically non-aligned part is processed by using a write-ahead log.

[0129] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0130] According to cache data in the cache data block corresponding to the pending write request and a preset decomposition threshold, the data corresponding to the pending write request is split into a logically aligned part and a logically non-aligned part.

[0131] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0132] Merge the cached data and the data corresponding to the pending write request to obtain merged data;

[0133] The data in the merged data whose data volume is greater than the preset decomposition threshold is determined as the logically aligned part, and the data in the merged data whose data volume is not greater than the preset decomposition threshold is determined as the logically non-aligned part.

[0134] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0135] A log entry for the logically non-aligned portion is generated, and the log entry is written to a log disk, and the logically non-aligned portion is written to a new cache data block.

[0136] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0137] If the pending write request is a write request of the second type, data corresponding to the pending write request is processed by using a pre-write log; the data volume of the second type of write request is not greater than the classification threshold.

[0138] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0139] Upon receiving a pending write request, determining the remaining space on the log disk;

[0140] Determine the classification threshold based on the remaining space on the log disk.

[0141] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0142] Determine the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request;

[0143] If the pending write request is a first type of write request, splitting data corresponding to the pending write request into a logically aligned part and a logically non-aligned part; and the data volume of the first type of write request is greater than the classification threshold;

[0144] The logically aligned part is written to the data disk and / or the logically non-aligned part is processed by using a write-ahead log.

[0145] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0146] According to cache data in the cache data block corresponding to the pending write request and a preset decomposition threshold, the data corresponding to the pending write request is split into a logically aligned part and a logically non-aligned part.

[0147] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0148] Merge the cached data and the data corresponding to the pending write request to obtain merged data;

[0149] The data in the merged data whose data volume is greater than the preset decomposition threshold is determined as the logically aligned part, and the data in the merged data whose data volume is not greater than the preset decomposition threshold is determined as the logically non-aligned part.

[0150] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0151] A log entry for the logically non-aligned portion is generated, and the log entry is written to a log disk, and the logically non-aligned portion is written to a new cache data block.

[0152] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0153] If the pending write request is a write request of the second type, data corresponding to the pending write request is processed by using a pre-write log; the data volume of the second type of write request is not greater than the classification threshold.

[0154] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0155] Upon receiving a pending write request, determining the remaining space on the log disk;

[0156] Determine the classification threshold based on the remaining space on the log disk.

[0157] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0158] Determine the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request;

[0159] If the pending write request is a first type of write request, splitting data corresponding to the pending write request into a logically aligned part and a logically non-aligned part; and the data volume of the first type of write request is greater than the classification threshold;

[0160] The logically aligned part is written to the data disk and / or the logically non-aligned part is processed by using a write-ahead log.

[0161] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0162] According to cache data in the cache data block corresponding to the pending write request and a preset decomposition threshold, the data corresponding to the pending write request is split into a logically aligned part and a logically non-aligned part.

[0163] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0164] Merge the cached data and the data corresponding to the pending write request to obtain merged data;

[0165] The data in the merged data whose data volume is greater than the preset decomposition threshold is determined as the logically aligned part, and the data in the merged data whose data volume is not greater than the preset decomposition threshold is determined as the logically non-aligned part.

[0166] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0167] A log entry for the logically non-aligned portion is generated, and the log entry is written to a log disk, and the logically non-aligned portion is written to a new cache data block.

[0168] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0169] If the pending write request is a write request of the second type, data corresponding to the pending write request is processed by using a pre-write log; the data volume of the second type of write request is not greater than the classification threshold.

[0170] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0171] Upon receiving a pending write request, determining the remaining space on the log disk;

[0172] Determine the classification threshold based on the remaining space on the log disk.

[0173] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0174] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0175] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A write request processing method, characterized in that: The method comprises: Determining the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request; If the pending write request is a write request of the first type, splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part; and the data volume of the first type of write request is greater than the classification threshold; The logically aligned portion is written into a data disk and / or the logically non-aligned portion is processed by using a write-ahead log.

2. The method according to claim 1, characterized in that The step of splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part includes: According to the cache data in the cache data block corresponding to the pending write request and a preset decomposition threshold, the data corresponding to the pending write request is split into a logically aligned part and a logically non-aligned part.

3. The method according to claim 2, characterized in that The step of splitting the data corresponding to the pending write request into a logically aligned part and a logically non-aligned part according to the cached data in the cached data block corresponding to the pending write request and a preset decomposition threshold comprises: Merging the cached data and the data corresponding to the pending write request to obtain merged data; The data in the merged data whose data volume is greater than the preset decomposition threshold is determined as the logically aligned part, and the data in the merged data whose data volume is not greater than the preset decomposition threshold is determined as the logically non-aligned part.

4. The method according to claim 1, characterized in that The processing of the logical non-aligned part by using a write-ahead log includes: A log entry of the logically non-aligned portion is generated, and the log entry is written into a log disk, and the logically non-aligned portion is written into a new cache data block.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: If the pending write request is a write request of the second type, data corresponding to the pending write request is processed by using a pre-write log method; and the data volume of the second type of write request is not greater than the classification threshold.

6. The method according to claim 1, characterized in that The method further comprises: Upon receiving the pending write request, determining remaining space on the log disk; The classification threshold is determined according to the remaining space of the log disk.

7. A write request processing device, characterized in that: The device comprises: A first determination module, configured to determine the type of the pending write request according to a preset classification threshold and the amount of data of the received pending write request; a splitting module, configured to split data corresponding to the pending write request into a logically aligned part and a logically non-aligned part if the pending write request is a first type of write request; and the data volume of the first type of write request is greater than the classification threshold; The first processing module is used to write the logically aligned part into a data disk and / or process the logically non-aligned part by using a write-ahead log.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images