Data writing method, apparatus, device, and medium

Abstract

The application discloses a data writing method, device, equipment and medium, and relates to the technical field of data storage. In the method, if target data in a data writing request is large, at least part of encoded data blocks obtained after encoding data blocks of the target data according to a first redundancy ratio is saved to at least one backup node of a cache pool. In this way, when part of the encoded data blocks is lost, the lost encoded data blocks can be recovered according to other non-lost encoded data blocks. Compared with a backup mode of multiple copies, the data redundancy storage mode based on erasure code can reduce storage resource occupation and bandwidth occupation in a data transmission process, thereby improving the performance of a storage system. Conversely, if the target data in the data writing request is small, the target data is still backed up in a multiple-copy mode. In this way, the time consumption of encoding calculation can be reduced, the time delay of the storage system is reduced, and the performance of the storage system is improved.

Description

Technical Field

[0001] This application relates to the field of data storage, and in particular to data writing methods, apparatus, devices, and media. Background Technology

[0002] To reduce performance pressure, some storage systems first save multiple small IO write requests to the cache layer, and then aggregate the multiple small IO write requests into a large IO write request before writing it to the storage pool.

[0003] Currently, in some technologies, after the caching layer receives a data write request, it typically uses a multi-replica approach to back up the data blocks from small I / O write requests across multiple cache nodes to prevent data loss. While this method can ensure data reliability, it can also lead to a decrease in storage system performance. Summary of the Invention

[0004] This application provides a data writing method, a data writing device, an electronic device, a computer-readable storage medium, and a computer program product to at least solve the problem of degraded storage system performance in related technologies.

[0005] This application provides a data writing method applied to the main cache node in a cache pool; including:

[0006] Receive a data write request, which includes the target data to be written to the storage pool;

[0007] If the data write request is a Type I data write request, then the target data is copied to at least one backup cache node in the cache pool to back up the target data;

[0008] If the data write request is a second type of data write request, the target data is divided into multiple data blocks, and the data blocks are encoded according to the first redundancy ratio. At least a portion of the encoded data blocks are then saved to at least one backup cache node in the cache pool to perform redundant storage of the target data. The target data size in the second type of data request is larger than the target data size in the first type of data request.

[0009] The system retrieves target data for multiple data write requests from the cache pool, aggregates the target data for multiple data write requests, and then writes the aggregated data to the storage pool.

[0010] This application also provides a data writing device, including:

[0011] The request receiving module is used to receive data write requests, which include the target data to be written to the storage pool.

[0012] The first data backup module is used to copy the target data to at least one backup cache node in the cache pool to back up the target data if the data write request is a first type of data write request.

[0013] The second data backup module is used to divide the target data into multiple data blocks if the data write request is a second type of data write request, and after encoding the data blocks according to the first redundancy ratio, save at least a portion of the encoded data blocks to at least one backup cache node in the cache pool to perform redundant storage of the target data. The target data size in the second type of data request is greater than the target data size in the first type of data request.

[0014] The data aggregation module is used to obtain the target data for multiple data write requests from the cache pool, aggregate the target data of multiple data write requests, and then write the aggregated data to the storage pool.

[0015] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above data writing methods when executing the computer program.

[0016] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data writing methods.

[0017] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data writing methods.

[0018] In some embodiments of this application, if the target data in the data write request is large, the data blocks of the target data are encoded according to a first redundancy ratio, and at least a portion of the encoded data blocks are saved to at least one backup cache node in the cache pool. Thus, if some encoded data blocks are lost, the lost encoded data blocks can be recovered based on the other unlost encoded data blocks. Compared to multi-replica backup methods, this erasure coding-based data redundancy storage method can reduce storage resource consumption and bandwidth consumption during data transmission, thereby improving storage system performance. Conversely, if the target data in the data write request is small, the target data is still backed up using a multi-replica method. This reduces the time spent on encoding calculations, lowers the latency of the storage system, and further improves storage system performance. In summary, the methods in some embodiments of this application can solve the problem of degraded storage system performance in related technologies. Attached Figure Description

[0019] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating the data writing method provided in this application embodiment;

[0021] Figure 2 A schematic diagram illustrating the process of target data recovery provided in this application embodiment;

[0022] Figure 3 A flowchart illustrating another data writing method provided in an embodiment of this application;

[0023] Figure 4 Schematic diagram of a data writing apparatus provided for some embodiments of this application;

[0024] Figure 5 A schematic diagram of the modules of an electronic device provided for some embodiments of this application. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0026] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0027] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] In storage systems, on the one hand, a large number of user-initiated I / O write requests will inevitably put significant performance pressure on the storage system. On the other hand, if too many operations need to be performed when users initiate I / O write requests, this will also inevitably put significant performance pressure on the storage system. For example, in a distributed storage system based on erasure coding, the user's original data is typically divided into K data blocks, and these data blocks are then encoded to obtain M parity blocks. These K+M data blocks and parity blocks are distributed across different storage nodes. For data accuracy, even if a user modifies one data block (e.g., data block A), all K+M data blocks and parity blocks need to be read, data block A modified, and the parity blocks recalculated based on the modified data block A and the other read data blocks. Finally, the new M parity blocks and K data blocks are written back to the storage nodes. This process introduces additional computation and data read / write operations, thus increasing the performance pressure on the storage system.

[0029] To reduce performance pressure on storage systems, some systems first cache multiple small I / O write requests and then aggregate them into a single large I / O write request before writing it to the storage pool. For example, in an erasure coding-based distributed storage system, if a user's first I / O write request requires modification of data block A, the modification of data block A can be paused, and the write-ahead log (WAL) records the data used to modify data block A (i.e., the new data to be written to the storage location of data block A) and the metadata information of data block A (such as the mapping between the logical address and physical address of data block A). If a user's second I / O write request requires modification of data block B, the first and second I / O write requests can be merged into a single large I / O write request based on the write-ahead log. The large I / O request specifies that data blocks A and B should be modified. Thus, when updating data blocks A and B in the storage node, only one data read, update, verification calculation, and write operation to the storage node is required, eliminating the need for two separate data read, update, verification calculation, and write operations. In this way, the performance pressure on the storage system can be reduced.

[0030] In some technologies, after the caching layer receives a data write request, to improve data storage reliability and prevent data loss, it typically employs a multi-replica approach, copying the write-ahead log used to record IO write requests to multiple cache nodes for backup. This copying of the write-ahead log to multiple cache nodes can include the following two operations:

[0031] 1) Copy the target data to be written to the storage node in the IO write request to multiple cache nodes;

[0032] 2) Copy the target data's metadata to multiple cache nodes.

[0033] While this approach can improve data storage reliability, it can also lead to a decrease in storage system performance. Simply put, replicating write-ahead logs to multiple cache nodes consumes bandwidth and storage resources. If there's a surge in I / O write requests, the demand for bandwidth and storage resources will increase dramatically, resulting in a performance degradation of the storage system.

[0034] In addition, in some technologies, solid-state drives (SSDs) are often used as cache pools. When write-ahead logs are copied to multiple cache nodes in a multi-replica manner, it will also exacerbate the wear and tear on the SSDs and shorten their lifespan.

[0035] In view of this, this application provides a data writing method that can solve the above problems. The data writing method can be applied to a storage system. The storage system may include a protocol layer, cache nodes, and a storage pool. The cache nodes may include a primary cache node and at least one backup cache node. The primary cache node and at least one backup cache node can serve as the aforementioned cache layer. The storage pool may include multiple storage nodes. The protocol layer may be the access layer of the storage system, for example, a service module in a distributed storage system that directly communicates with clients, receives and parses requests.

[0036] See also Figure 1 This is a flowchart illustrating a data writing method provided in some embodiments of this application. Figure 1 In this context, the data writing method includes the following steps:

[0037] Step S101: Receive a data write request, which includes the target data to be written to the storage pool.

[0038] Specifically, the protocol layer in the storage system is used to receive data write requests. For example, when a user needs to save data D1 to the storage system, they can interact with the protocol layer through a client and send a data write request to the protocol layer. The data write request can include data D1 and its logical storage address, address1. As another example, after saving data D1 to the storage system, if the user needs to modify data D1 to data D2, they can interact with the protocol layer again through a client and send a data write request to the protocol layer. This data write request can also include data D2 and its logical storage address, address1.

[0039] After receiving a data write request at the protocol layer, the request type can be determined based on the target data size in the request. Specifically, the request type can include a first type of data write request and a second type of data write request. The target data size in the second type of data write request is larger than the target data size in the first type of data write request.

[0040] In this embodiment, the first type of data write request refers to a request where the target data size is less than or equal to a first threshold. The second type of data write request refers to a request where the target data size is greater than the first threshold but less than or equal to a second threshold. The first threshold is less than the second threshold. For example, assume the first threshold is 32KB and the second threshold is 256KB. If the target data size in the data write request is less than or equal to 32KB, the data write request is determined to be a first type of data write request; if the target data size in the data write request is greater than 32KB but less than or equal to 256KB, the data write request is determined to be a second type of data write request. When the data write request is either the first type or the second type of data write request, the protocol layer can send the data write request to the main cache node. The main cache node then continues to execute steps S102 to S103.

[0041] Step S102: If the data write request is a first type of data write request, then the target data is copied to at least one backup cache node in the cache pool to back up the target data.

[0042] Specifically, when the data write request is a Type I data write request, the primary cache node can execute a multi-replica backup strategy (such as a three-replica backup strategy). Based on the multi-replica backup strategy, the primary cache node can save the target data in the data write request locally and continue to replicate the target data to at least one backup cache node, thus achieving multi-replica backup of the target data.

[0043] Step S103: If the data write request is a second type of data write request, the target data is divided into multiple data blocks, and the data blocks are encoded according to the first redundancy ratio. At least some of the encoded data blocks are then saved to at least one backup cache node in the cache pool to perform redundant storage of the target data.

[0044] Specifically, when the data write request is the second data write request, the primary cache node can divide the target data in the data write request into multiple fixed-size data blocks, and encode the data blocks using a first redundancy ratio to obtain at least one check block. The data blocks and check blocks are collectively referred to as encoded data blocks. These encoded data blocks are then saved to different backup cache nodes in the cache pool to achieve redundant storage of the target data.

[0045] For example, the primary cache node can use erasure coding with a redundancy ratio of 4:2 to encode the target data, dividing it into four data blocks. Two parity blocks are then calculated based on these four blocks, resulting in six encoded data blocks. These six encoded data blocks can be stored across six different cache nodes, including the primary cache node. Specifically, one encoded data block is stored on the primary cache node, and the remaining five are stored on five different backup cache nodes. Alternatively, the six encoded data blocks can be stored on six different backup cache nodes. This ensures that if any two or fewer cache nodes containing the encoded data blocks fail, data recovery can still be achieved using the encoded data blocks from the remaining cache nodes, guaranteeing data storage reliability.

[0046] In this embodiment, when the target data size in the data write request is small, the bandwidth and storage resources consumed are relatively small. Therefore, by using multiple replica backups, the encoding calculation time can be reduced, the latency of the storage system can be lowered, and the performance of the storage system can be improved. When the target data size in the data write request is large, the bandwidth and storage resources consumed are relatively large. Therefore, after encoding the data blocks of the target data according to the first redundancy ratio, at least a portion of the encoded data blocks can be saved to at least one backup cache node in the cache pool to achieve redundant storage and data recovery of the target data. Compared with the multiple replica backup method, this erasure coding-based data redundancy storage method can reduce storage resource consumption and bandwidth consumption during data transmission, thereby improving the performance of the storage system.

[0047] For example, with a target data size of 16KB, a three-replica backup method only requires 48KB of bandwidth and storage resources (i.e., relatively few resources). Therefore, encoding the data blocks is unnecessary, saving encoding time and improving storage system performance. However, with a target data size of 64KB, a three-replica backup method requires 192KB of bandwidth and storage resources (i.e., significantly more resources). Therefore, a 4:2 erasure coding method can be used to encode the target data. After encoding, the resulting six encoded data blocks only require 96KB of storage resources, thus saving half of the bandwidth and storage resources and further improving storage system performance.

[0048] Step S104: Obtain the target data for multiple data write requests from the cache pool, aggregate the target data for multiple data write requests, and then write the aggregated data to the storage pool.

[0049] Specifically, after multiple data write requests are written to the cache pool, the main cache node performs a periodic aggregation operation on the target data of the data write requests stored in the cache pool, merging the target data of multiple data write requests into an aggregated data block, and writing it to the storage node in the storage pool.

[0050] In summary, the data writing method provided in this embodiment, if the target data in the data writing request is large, encodes the data blocks of the target data according to the first redundancy ratio, and saves at least a portion of the encoded data blocks to at least one backup cache node in the cache pool. Thus, if some encoded data blocks are lost, the lost encoded data blocks can be recovered based on the other unlost encoded data blocks. Compared to multi-replica backup methods, this erasure coding-based data redundancy storage method can reduce storage resource consumption and bandwidth consumption during data transmission, thereby improving storage system performance. Conversely, if the target data in the data writing request is small, the target data is still backed up using a multi-replica approach. This reduces the time spent on encoding calculations, lowers the latency of the storage system, and further improves storage system performance. In summary, the methods of some embodiments of this application can solve the problem of degraded storage system performance in related technologies.

[0051] In some embodiments, the request category may further include a third type of data write request. The target data size in a third type of data write request is larger than the target data size in a second type of data write request. Specifically, a third type of data write request refers to a request whose target data size is larger than a second threshold. When a data write request is a third type of data write request, the target data in the data write request is equivalent to the size of an aggregated data block, meaning there is no need for the target data in the data write request to be aggregated with other data. Therefore, when the protocol layer determines that a data write request is a third type of data write request based on the target data size in the data write request, it can directly write the target data specified in the data write request to the storage pool without going through the cache pool. This avoids excessive consumption of cache resources.

[0052] In some embodiments, writing aggregated data or target data to a storage pool includes:

[0053] The aggregated data or target data is divided into multiple data blocks, and the data blocks are encoded according to the second redundancy ratio. The resulting encoded data blocks are then saved to the storage pool, where the redundancy of the second redundancy ratio is less than that of the first redundancy ratio.

[0054] Specifically, the main cache node is used to divide the aggregated data of the first type of data request and the second type of data request into multiple data blocks, and after encoding the data blocks according to the second redundancy ratio, save the resulting encoded data blocks to the storage pool.

[0055] The protocol layer is used to divide the target data of the third type of data write request into fixed-size data blocks, and encode the above data blocks using a second redundancy ratio to calculate at least one check block.

[0056] Data blocks and check blocks are collectively referred to as encoded data blocks, which are stored in the storage nodes of the storage pool. The redundancy of the second redundancy ratio is less than the redundancy of the first redundancy ratio described in step S103 above. This reduces storage resource consumption.

[0057] For example, assuming the first redundancy ratio used in step S103 above is 4:2, the second redundancy ratio can be 16:2. The first redundancy degree of the first redundancy ratio is 0.5, and the second redundancy degree of the second redundancy ratio is 0.125. Since the redundancy degree of the second redundancy ratio is less than that of the first redundancy ratio, storage overhead can be significantly reduced and the utilization rate of storage resources can be improved while ensuring data reliability.

[0058] In some embodiments, after writing the aggregated data or target data into the storage pool as described above, the method further includes the following steps a1 to a2.

[0059] Step a1: After saving the encoded data block to the storage pool, establish a mapping relationship between the physical storage address of the encoded data block and the logical address of each target data corresponding to the encoded data block.

[0060] Specifically, a physical storage address refers to the storage address of a coded data block in each storage node, while the logical address of each target data corresponding to the coded data block is a virtual address. Multiple non-contiguous physical storage addresses can be mapped to a contiguous logical address, thereby improving the utilization efficiency of storage resources.

[0061] Step a2: Upon receiving a data read request, determine the physical storage address of the data to be read based on the mapping relationship.

[0062] Specifically, after establishing the mapping relationship, it can be recorded in an address mapping table. When a data read request is received, the corresponding physical storage address can be found in the address mapping table based on the logical address of the target data contained in the request, thereby locating the storage node where the target data is located and obtaining the corresponding encoded data block.

[0063] In this embodiment, if the target data to be read is stored using erasure coding with a redundancy ratio of 16:2, then the corresponding encoded data blocks need to be read from the storage nodes to restore the data. If all the storage nodes storing the encoded data blocks are functioning correctly, only 16 data blocks need to be read to restore the original data. Typically, storage nodes with lower loads are prioritized for reading the encoded data blocks. If there are faulty nodes, and the number does not exceed two, then the encoded data blocks from the healthy storage nodes need to be read, and data recovery is achieved through decoding. If more than two storage nodes are faulty, the original data cannot be recovered from the existing encoded data blocks. In this case, error information can be logged, and an error message indicating that the data is unrecoverable can be returned to the upper-layer application.

[0064] In the above embodiments, by establishing a mapping relationship between the physical storage address of the encoded data block and the logical address of each target data corresponding to the encoded data block, when a data read request is received, the storage node where the target data is located can be quickly located and the required target data can be obtained. This avoids the low data read efficiency caused by traversing and searching all storage nodes, thereby significantly reducing the data read latency and improving the system read performance.

[0065] In some embodiments, after the mapping relationship is created, the mapping relationship is saved to at least one backup cache node in the local cache pool and the cache pool to back up the mapping relationship.

[0066] Specifically, after the mapping relationship between the logical address and physical address of the target data is created, the mapping relationship can be saved to the local database of the primary cache node and replicated to at least one backup cache node through a multi-replica mechanism to achieve persistent storage of the mapping relationship.

[0067] Furthermore, while backing up the mapping relationship through the multi-replica mechanism, the basic attributes of the target data, such as data size and creation time, i.e. the metadata of the target data, can also be saved to the local database of the primary cache node and at least one backup cache node, thus realizing the persistent storage of metadata through multiple replicas.

[0068] In the above embodiments, the mapping relationship is stored in the local database of the primary cache node through a multi-replica mechanism and then synchronized to multiple backup cache nodes. This ensures that even if the primary cache node fails, the mapping relationship can still be obtained through the backup cache nodes, thus guaranteeing the reliability of data reading.

[0069] In some embodiments, the data writing method may further include steps b1 to b2.

[0070] Step b1: After receiving a data write request, if the data write request is a first type of data write request or a second type of data write request, then assign a pre-write log sequence number to the data write request. The pre-write log sequence number is different for different data write requests.

[0071] Specifically, the pre-write log sequence number is used to indicate the order in which data write requests are made. When a data write request is a first-type or second-type data write request, the primary cache node assigns a unique and incrementing pre-write log sequence number to the request to identify the order of the write operations.

[0072] For example, if the target data size range for the first type of data write request is set to less than or equal to 32KB, and the target data size range for the second type of data write request is set to greater than 32KB and less than or equal to 256KB, then when the main cache node receives a data write request with a target data size less than or equal to 256KB, it will assign a pre-write log sequence number to it.

[0073] It is important to note that when the first type of data write request is persisted through the multi-replica mechanism, although there are three first type of data write requests, these three first type of data write requests are essentially the same first type of data write request. That is, the log sequence numbers of these first type of data write requests on different nodes are consistent.

[0074] Similarly, when the second data write request obtains multiple encoded data blocks through the first redundancy ratio encoding, these encoded data blocks all belong to a single data write request, that is, these encoded data blocks share the same pre-write log sequence number.

[0075] Step b2: Aggregate the target data of multiple data write requests according to the allocation time order of the pre-write log.

[0076] Specifically, when aggregating the target data of multiple data write requests in the main cache node, since the pre-write log sequence number allocated through step b1 above is unique and incrementing, multiple data write requests can be sorted and aggregated according to the size of the pre-write log sequence number, or multiple data write requests can be aggregated according to the allocation time order of the pre-write log sequence number, ensuring that the aggregation operation strictly follows the write time sequence.

[0077] In addition, the primary cache node can aggregate data write requests without relying on the pre-write log sequence number. For example, it can aggregate data write requests sequentially according to the first-in-first-out principle through the node's internal memory queue, ensuring that the first-arriving requests are processed first.

[0078] In the above embodiments, when a data write request is determined to be of type 1 or type 2, a unique pre-write log sequence number is assigned to it. Then, multiple data write requests are sorted and aggregated based on the temporal relationship of these pre-write log sequence numbers. This avoids data inconsistency caused by disordered write order and ensures that the order of data writing and request processing are consistent.

[0079] In some embodiments, step b2, which aggregates the target data of multiple data write requests according to the allocation time order of the pre-write log sequence number, includes the following:

[0080] If the target data of the first data write request and the second data write request need to be written to the same physical storage address of the storage pool, and the pre-write log sequence number of the first data write request is allocated before the pre-write log sequence number of the second data write request, then the target data of the first data write request is deleted, and the target data of the second data write request is aggregated with the target data of other data write requests.

[0081] Specifically, if the target data of the first data write request and the second data write request need to be written to the same physical storage address, then the target data needs to be aggregated in the main cache node. If the pre-write log sequence number of the first data write request is allocated before the pre-write log sequence number of the second data write request, then the target data of the second data write request is retained, and the target data of the first data write request is deleted, ensuring that the data written later can overwrite the data written earlier. If there are other data write requests whose target data still need to be aggregated, then the comparison continues according to the order of the pre-write log sequence numbers, and the aggregation operation is performed in the main cache node.

[0082] The first data write request and the second data write request can be any two requests that conflict with writing to the same physical address. That is, they can both be first type data write requests, both be second type data write requests, or one can be a first type data write request and the other is a second type data write request.

[0083] The description of this embodiment can be further illustrated by the following example. For instance, when a client initiates two Type I data write requests, the pre-write log sequence number of the first request A is 1001, and the pre-write log sequence number of the second request B is 1002. The target data "1111" of request A and the target data "2222" of request B need to be written to the same physical storage address. In this case, according to the order of the pre-write log sequence numbers, the target data "1111" of request A will be deleted, and the target data "2222" of request B will be retained. If there are no other requests with the same physical address write requirement, then the target data "2222" will eventually be written to that physical address.

[0084] In the above embodiments, for target data that needs to be written to the same physical storage address, the aggregation strategy provided in this embodiment effectively reduces the residence of redundant data in the cache and improves storage resource utilization while ensuring the consistency of data write timing. Furthermore, by executing the overwrite logic according to the allocation order of the pre-write log sequence number, it is ensured that data written later always overwrites data written earlier, avoiding subsequent read errors caused by stale data residue.

[0085] In some embodiments, the aggregation of target data for multiple data write requests based on the allocation time order of the pre-write log sequence number in step b2 further includes the following.

[0086] If the target data of the first data write request includes a first part of target data and a second part of target data, and if the first part of target data and the target data of the second data write request need to be written to the same physical storage address of the storage pool, and the pre-write log sequence number of the first data write request is allocated before the pre-write log sequence number of the second data write request, then the first part of the target data of the first data write request is deleted, and the second part of the target data of the first data write request is aggregated with the target data of the second data write request.

[0087] Specifically, if the target data of the first data write request consists of two parts, and the first part of the target data and the target data of the second data write request need to be written to the same physical storage address, then an aggregation operation is also required on the main cache node. If the pre-write log sequence number of the first data write request is allocated before the pre-write log sequence number of the second data write request, then the first part of the target data in the first data write request is deleted, the second part of the target data is retained, and the second part of the target data is aggregated with the target data of the second data write request.

[0088] Similarly, the first and second data write requests mentioned above do not limit the type of data write request. This aggregation rule applies to both types of data write requests.

[0089] The description of this embodiment can be further illustrated by the following example. For instance, when a client initiates a first type of data write request and a second type of data write request, the first request A is a second type of data write request with a pre-write log sequence number of "1001", containing two target data parts: A1 "1111" and A2 "2222"; the second request B is a first type of data write request with a pre-write log sequence number of "1002", and its target data is "3333". The target data A1 and the target data B need to be written to the same physical storage address C1, while the target data A2 needs to be written to physical storage address C2. At this time, according to the order of the pre-write log sequence numbers, the target data A1 "1111" will be deleted, and the target data "3333" of request B will be retained and aggregated with the target data A2 "2222" on the main cache node. If no other data write requests subsequently overwrite physical addresses C1 and C2, then the data "3333" will ultimately be stored in physical storage address C1, and the data "2222" will be stored in physical storage address C2.

[0090] In the above embodiments, through the aggregation strategy provided in this embodiment, for multiple data write requests of different target data sizes, the main cache node can perform aggregation operations on the target data based on the pre-write log sequence number and physical storage address, which significantly improves the system's flexibility in dealing with target data of different sizes.

[0091] In some embodiments, after assigning a pre-write log sequence number, the data writing method includes the following steps c1 to c2.

[0092] Step c1: Create a pre-write log corresponding to each data write request. The pre-write log for each data write request includes the pre-write log sequence number of the corresponding data write request.

[0093] Step c2 involves saving the pre-write log to at least one backup cache node in the local cache pool and the cache pool to back up the pre-write log.

[0094] Specifically, the write-ahead log is the aforementioned write-ahead log. After the primary cache node assigns a write-ahead log sequence number, it creates a corresponding write-ahead log for each data write request and records the write-ahead log in the generated write-ahead log. In addition, the write-ahead log can also record the target data of the request and the metadata of the target data (such as the physical storage address of the target data and the request type).

[0095] Referring to steps S102 to S103 above, when the primary cache node performs persistent storage of target data for different types of data write requests, different strategies can be selected for processing. Similarly, when the pre-write log needs to be saved to the primary cache node and at least one backup cache node, different persistence strategies can also be selected for processing.

[0096] In the above embodiments, by creating a pre-write log corresponding to each data write request and persistently backing it up on the main cache node and at least one backup cache node, the main cache node can track and replay the logs using the pre-write log sequence number. Furthermore, it can ensure the recoverability of the target data when some cache nodes fail, thereby improving the reliability of the system.

[0097] In some embodiments, for any data write request, if the data write request is a first type of data write request or a second type of data write request, then when writing the target data in the data write request to the storage pool, the pre-write log corresponding to the data write request is deleted in the local and backup cache nodes.

[0098] Specifically, after the primary cache node performs an aggregation operation on either the first type of data write request or the second type of data write request, it writes the target data to the storage pool for long-term persistent storage. Then, the pre-write logs corresponding to the data write requests stored in both the primary and backup cache nodes are deleted.

[0099] In this embodiment, by synchronously deleting the corresponding pre-write logs in the cache nodes after the storage pool is completed, some cache space in the cache pool can be released, avoiding the waste of cache resources caused by the accumulation of pre-write logs, thereby improving the space utilization and stability of the cache pool.

[0100] In some embodiments, in the event of a primary cache node failure, one of the backup cache nodes is used as the new primary cache node. The new primary cache node determines, based on the pre-write log, at least one data write request for which the target data has not yet been written to the storage pool, aggregates the target data of the at least one data write request, and writes the aggregated data to the storage pool.

[0101] Specifically, if the current primary cache node fails, any standby cache node in the cache pool will be used as the new primary cache node. The new primary cache node will parse the locally stored pre-write logs, aggregate the target data that has not been written to the storage pool, and write the aggregated data to the storage pool.

[0102] For example, when the cache pool contains both the target data for a first type of data write request that has not yet been written to the storage pool and the target data for a second type of data write request, refer to [reference needed]. Figure 2 , Figure 2 The flowchart for target data recovery provided in this application embodiment shows that when a backup cache node acts as a new primary cache node, the new primary cache node implements the following steps S201 to S204.

[0103] Step S201: Determine the request type.

[0104] Step S202: Restore the target data in the pre-write log.

[0105] Step S203: Determine whether all logs before writing have been read and parsed.

[0106] Step S204: Aggregate the target data based on the pre-write log sequence number.

[0107] Specifically, for the first type of data write request, the new primary cache node reads the locally stored copy of the pre-write log and parses the target data, physical storage address, and other information recorded therein. For the second type of data write request, the new primary cache node reads the locally stored pre-write log containing encoded data blocks and decodes it to restore the target data for the second type of data write request. Once all locally stored pre-write logs have been read and parsed, the new primary cache node aggregates the target data for multiple data write requests based on the pre-write log sequence number.

[0108] In this embodiment, the parsing and recording of the pre-write log can be implemented using a hash table. The business identifier corresponding to the data write request is recorded by key-value pairs, and the pre-write log is recorded by a linked list. Each node of the linked list stores the pre-write log that has been read.

[0109] In the above embodiments, if the current primary cache node fails, a backup cache node is selected as the new primary cache node. The new primary cache node writes the target data of the write request based on the locally stored pre-write log aggregation data and writes it into the storage pool. This ensures that even after a failure, the uncommitted data write request can still be accurately recovered, thus guaranteeing data consistency and business continuity.

[0110] In some embodiments, the aggregation of target data for multiple data write requests in step S104 includes the following:

[0111] Aggregate multiple first-type data into the target data of the request;

[0112] And / or, aggregate multiple second-type data into the target data of the request;

[0113] And / or, aggregate the target data of at least one first-class data write request and at least one second-class data write request.

[0114] Specifically, after receiving multiple data write requests, the primary cache node persists the data to the cache pool using different write strategies based on the target data size of the requests, and then performs an aggregation operation. During aggregation, the primary cache node can aggregate any number of first-type and second-type data write requests, considering only the pre-write log sequence number or the order in the memory queue.

[0115] For example, when the primary cache node receives and processes 1MB of target data requests, an aggregation operation is triggered. For the 1MB target data volume, it can include only the first type of data write requests, only the second type of data write requests, or both types of data write requests. Afterward, the primary cache node encodes the aggregated target data using a low redundancy ratio and writes the encoded data blocks to different storage nodes in the storage pool.

[0116] In the above embodiments, the aggregation operation of the main cache node is not limited to a single type of data write request, thereby improving the flexibility and storage efficiency of data aggregation and adapting to the write load under different business scenarios.

[0117] Based on the description of the above embodiments, the different persistence strategies adopted for the pre-write log can be further illustrated by the following examples.

[0118] In this embodiment, a data writing method is provided, which can be referred to in conjunction with the following: Figure 3 , Figure 3 This is a flowchart illustrating another data writing method provided in an embodiment of this application. Figure 3 The data writing method also includes the following steps.

[0119] Step S301: Receive a data write request. See details below. Figure 1 Step S101 of the illustrated embodiment.

[0120] In addition, it should be noted that in this embodiment, before the protocol layer issues a data write request, it can make a judgment on the target data size of the data write request based on a preset threshold to ensure that large-scale data write requests exceeding the preset threshold, i.e., third-type data write requests, do not enter the cache pool.

[0121] For example, if a user pre-sets a threshold of 256KB, when the target data size of a data write request prepared by the protocol layer is less than or equal to 256KB, the protocol layer will send the data write request to the main cache node. If the target data size of the data write request is greater than 256KB, the protocol layer will determine that the data write request is a third-type data write request and will not send the request but will send it directly to the storage node of the storage pool.

[0122] Step S302: Assign a pre-write log sequence number.

[0123] Specifically, since the protocol layer has excluded the third type of data write request from the cache pool in step S301, a unique and incremental pre-write log sequence number needs to be assigned to all data write requests from the protocol layer received by the main cache node.

[0124] Step S303: Determine the size of the target data requested.

[0125] Specifically, in the main cache node, the target data size of the data write request is determined again, and the request is divided into the first type of data write request and the second type of data write request.

[0126] Step S304: If it is a first-type data write request, generate a pre-write log and replicate it to the other two backup cache nodes. For details, please refer to [link to relevant documentation]. Figure 1 Step S102 of the illustrated embodiment.

[0127] It should be noted that the target data in step S102 has already been written to the generated pre-write log. Operations related to the target data in this step are equivalent to operations on the pre-write log.

[0128] Step S305: If it is a second type of data write request, then an erasure coding operation is performed, and the resulting encoded data blocks are stored in different cache nodes. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment.

[0129] It is important to note that the encoded data block obtained through erasure coding in this step has also been written to the pre-write log. Operations on the encoded data block are equivalent to operations on the pre-write log.

[0130] Step S306: Return the response indicating that the pre-write log persistence storage is complete.

[0131] Specifically, after the primary cache node completes saving the pre-write log in the cache pool, it returns the result to the protocol layer.

[0132] Step S307: Aggregate the data and save it to the storage pool. See details below. Figure 1 Step S104 of the illustrated embodiment.

[0133] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0134] Corresponding to the data writing method, this application also provides a data writing device. (See also...) Figure 4 This is a schematic diagram of a data writing device provided in some embodiments of this application. Figure 4 The data writing device includes...

[0135] The request receiving module 401 is used to receive data write requests, which include target data to be written to the storage pool.

[0136] The first data backup module 402 is used to copy the target data to at least one backup cache node in the cache pool to back up the target data if the data write request is a first type of data write request.

[0137] The second data backup module 403 is used to divide the target data into multiple data blocks if the data write request is a second type of data write request, and after encoding the data blocks according to the first redundancy ratio, save at least a portion of the encoded data blocks to at least one backup cache node in the cache pool to perform redundant storage of the target data. The target data size in the second type of data request is greater than the target data size in the first type of data request.

[0138] The data aggregation module 404 is used to obtain the target data of multiple data write requests from the cache pool, aggregate the target data of multiple data write requests, and then write the aggregated data to the storage pool.

[0139] In some embodiments, the data writing method further includes:

[0140] The data writing module is used to write the target data specified in the data writing request directly to the storage pool without going through the cache pool if the data writing request is a third type of data writing request. The target data size in the third type of data request is larger than the target data size in the second type of data request.

[0141] In some embodiments, the data writing module is further configured to:

[0142] The aggregated data or target data is divided into multiple data blocks, and the data blocks are encoded according to the second redundancy ratio. The resulting encoded data blocks are then saved to the storage pool, where the redundancy of the second redundancy ratio is less than that of the first redundancy ratio.

[0143] In some embodiments, the data writing module is further configured to:

[0144] After saving the encoded data block to the storage pool, a mapping relationship is established between the physical storage address of the encoded data block and the logical address of each target data corresponding to the encoded data block;

[0145] Upon receiving a data read request, the physical storage address of the data to be read is determined based on the mapping relationship.

[0146] In some embodiments, the data writing module is further configured to:

[0147] After creating the mapping relationship, save the mapping relationship to at least one backup cache node in the local cache pool and the cache pool to back up the mapping relationship.

[0148] In some embodiments, the request receiving module 401 is further configured to:

[0149] Upon receiving a data write request, if the data write request is a first type of data write request or a second type of data write request, a pre-write log sequence number is assigned to the data write request. The pre-write log sequence number is different for different data write requests.

[0150] Based on the order of the allocation time of the pre-write log sequence number, the target data of multiple data write requests are aggregated.

[0151] In some embodiments, the target data of multiple data write requests are aggregated according to the allocation time order of the pre-write log sequence number. The request receiving module 401 is further configured to:

[0152] If the target data of the first data write request and the second data write request need to be written to the same physical storage address of the storage pool, and the pre-write log sequence number of the first data write request is allocated before the pre-write log sequence number of the second data write request, then the target data of the first data write request is deleted, and the target data of the second data write request is aggregated with the target data of other data write requests.

[0153] In some embodiments, the target data of multiple data write requests are aggregated according to the allocation time order of the pre-write log sequence number. The request receiving module 401 is further configured to:

[0154] If the target data of the first data write request includes a first part of target data and a second part of target data, and if the first part of target data and the target data of the second data write request need to be written to the same physical storage address of the storage pool, and the pre-write log sequence number of the first data write request is allocated before the pre-write log sequence number of the second data write request, then the first part of target data of the first data write request is deleted, and the second part of target data of the first data write request is aggregated with the target data of the second data write request.

[0155] In some embodiments, after assigning the pre-write log sequence number, the request receiving module 401 is further configured to:

[0156] Create a pre-write log corresponding to each data write request. The pre-write log for each data write request includes the pre-write log sequence number of the corresponding data write request.

[0157] Save the pre-write log to at least one backup cache node in the local cache pool and the cache pool to back up the pre-write log.

[0158] In some embodiments, the data writing method further includes:

[0159] The log deletion module is used to delete the pre-write log corresponding to the data write request in both the local machine and the backup cache node when writing the target data in the data write request to the storage pool, if the data write request is a first type of data write request or a second type of data write request.

[0160] In some embodiments, the data aggregation module 404 is further configured to:

[0161] In the event of a primary cache node failure, one of the backup cache nodes is appointed as the new primary cache node. The new primary cache node uses the pre-write log to identify at least one data write request for which the target data has not yet been written to the storage pool. After aggregating the target data of the at least one data write request, the new primary cache node writes the aggregated data to the storage pool.

[0162] In some embodiments, the data aggregation module 404 is further configured to:

[0163] Aggregate multiple first-type data into the target data of the request;

[0164] And / or, aggregate multiple second-type data into the target data of the request;

[0165] And / or, aggregate the target data of at least one first-class data write request and at least one second-class data write request.

[0166] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0167] See also Figure 5 Embodiments of this application also provide an electronic device including a memory 10 and a processor 20, wherein the memory 10 stores a computer program and the processor 20 is configured to run the computer program to perform the steps in any of the above-described data writing method embodiments.

[0168] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data writing method embodiments when running.

[0169] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0170] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data writing method embodiments.

[0171] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data writing method embodiments.

[0172] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] The data writing method, apparatus, device, and medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data writing method, characterized in that, The method is applied to the primary cache node in the cache pool; the method includes: Receive a data write request, the data write request including target data to be written to the storage pool; If the data write request is a first type of data write request, then the target data is copied to at least one backup cache node in the cache pool to back up the target data; If the data write request is a second type of data write request, the target data is divided into multiple data blocks, and after encoding the data blocks according to the first redundancy ratio, at least a portion of the encoded data blocks are saved to at least one backup cache node of the cache pool to perform redundant storage of the target data. The target data size in the second type of data request is greater than the target data size in the first type of data request. The target data for multiple data write requests is obtained from the cache pool, and after aggregating the target data for the multiple data write requests, the aggregated data is written to the storage pool. Furthermore, upon receiving the data write request, if the data write request is either the first type of data write request or the second type of data write request, a pre-write log sequence number is assigned to the data write request, and the pre-write log sequence number is different for different data write requests. Based on the order of the allocation time of the pre-write log sequence number, aggregate the target data of multiple data write requests; The aggregation of target data from multiple data write requests based on the allocation time order of the pre-write log sequence number includes: If the target data of the first data write request includes a first part of target data and a second part of target data, and if the first part of target data and the target data of the second data write request need to be written to the same physical storage address of the storage pool, and the pre-write log sequence number of the first data write request is allocated before the pre-write log sequence number of the second data write request, then the first part of target data of the first data write request is deleted, and the second part of target data of the first data write request is aggregated with the target data of the second data write request.

2. The method according to claim 1, characterized in that, The method further includes: If the data write request is a third type of data write request, then the target data specified in the data write request is directly written to the storage pool without going through the cache pool. The target data size in the third type of data request is greater than the target data size in the second type of data request.

3. The method according to claim 2, characterized in that, Writing the aggregated data or the target data into the storage pool includes: The aggregated data or the target data is divided into multiple data blocks, and the data blocks are encoded according to a second redundancy ratio. The resulting encoded data blocks are then saved to the storage pool, wherein the redundancy of the second redundancy ratio is less than the redundancy of the first redundancy ratio.

4. The method according to claim 3, characterized in that, The method further includes: After saving the encoded data block to the storage pool, a mapping relationship is established between the physical storage address of the encoded data block and the logical address of each target data corresponding to the encoded data block; Upon receiving a data read request, the physical storage address of the data to be read is determined based on the mapping relationship.

5. The method according to claim 4, characterized in that, The method further includes: After the mapping relationship is created, it is saved locally and to at least one backup cache node of the cache pool to back up the mapping relationship.

6. The method according to claim 1, characterized in that, The aggregation of target data from multiple data write requests based on the allocation time order of the pre-write log sequence number includes: If the target data of the first data write request and the second data write request need to be written to the same physical storage address of the storage pool, and the pre-write log sequence number of the first data write request is allocated before the pre-write log sequence number of the second data write request, then the target data of the first data write request is deleted, and the target data of the second data write request is aggregated with the target data of other data write requests.

7. The method according to claim 1, characterized in that, After assigning the pre-write log sequence number, the method further includes: Create a pre-write log corresponding to each of the data write requests, wherein the pre-write log for each data write request includes the pre-write log sequence number of the corresponding data write request; The pre-write log is saved locally and to at least one backup cache node in the cache pool to back up the pre-write log.

8. The method according to claim 7, characterized in that, The method further includes: For any of the data write requests, if the data write request is the first type of data write request or the second type of data write request, then when writing the target data in the data write request to the storage pool, the pre-write log corresponding to the data write request is deleted in the local machine and the backup cache node.

9. The method according to claim 8, characterized in that, In the event of a failure of the primary cache node, one of the backup cache nodes is appointed as the new primary cache node. The new primary cache node determines, based on the pre-write log, at least one data write request for which the target data has not yet been written to the storage pool, aggregates the target data of the at least one data write request, and writes the aggregated data to the storage pool.

10. The method according to claim 1, characterized in that, The aggregation of the target data for the multiple data write requests includes: Aggregate multiple first-type data into the target data of the request; And / or, aggregate multiple second-type data into the target data of the request; And / or, aggregate the target data of at least one first-class data write request and at least one second-class data write request.

11. A data writing device, characterized in that, The device includes: A request receiving module is used to receive a data write request, wherein the data write request includes target data to be written to the storage pool; The first data backup module is used to copy the target data to at least one backup cache node in the cache pool if the data write request is a first type of data write request, so as to back up the target data. The second data backup module is used to divide the target data into multiple data blocks if the data write request is a second type of data write request, and after encoding the data blocks according to the first redundancy ratio, save at least a portion of the encoded data blocks to at least one backup cache node of the cache pool to perform redundant storage of the target data. The target data size in the second type of data request is greater than the target data size in the first type of data request. The data aggregation module is used to obtain target data for multiple data write requests from the cache pool, aggregate the target data of the multiple data write requests, and write the aggregated data into the storage pool. Specifically, after receiving the data write request, if the data write request is either the first type of data write request or the second type of data write request, a pre-write log sequence number is assigned to the data write request. Different data write requests have different pre-write log sequence numbers. The target data of multiple data write requests are aggregated according to the allocation time order of the pre-write log sequence numbers. Specifically, if the target data of the first data write request includes a first part of target data and a second part of target data, and if the first part of target data and the target data of the second data write request need to be written to the same physical storage address in the storage pool, and the pre-write log sequence number of the first data write request is allocated before the pre-write log sequence number of the second data write request, then the first part of target data of the first data write request is deleted, and the second part of target data of the first data write request is aggregated with the target data of the second data write request.

12. An electronic device, characterized in that, include: Memory: Used to store computer programs; A processor, configured to implement the steps of the data writing method as described in any one of claims 1 to 10 when executing the computer program.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data writing method as described in any one of claims 1 to 10.

Images