Data writing method, electronic device, storage medium, and program product

Abstract

The application discloses a data writing method, an electronic device, a storage medium and a program product, and relates to the technical field of storage, and comprises the following steps: when a write request is received each time, the write request is not directly written into a storage device, but log and operation information can be recorded first, and then, when it is determined that a trigger condition is met, data flushing operation is performed from front to back according to address information. In this process, the data writing is sequential writing, and the influence of random writing on storage performance can be avoided. In addition, in the case of accidental power failure, since the data has been stored in the log area of the storage device, data loss does not occur. Moreover, since the internal storage space of the sub-log area is continuous, the log is sequentially and incrementally written according to the order of the received write request, random writing is not needed, and the influence of random writing on storage performance can be reduced.

Description

Technical Field

[0001] This application relates to the field of storage technology, and in particular to data writing methods, electronic devices, storage media, and program products. Background Technology

[0002] With the current acceleration of digitalization and the explosive growth of global data, traditional centralized storage can hardly meet the demands in terms of scalability, performance, and reliability. Distributed storage systems have become mainstream due to their advantages of multi-node parallel processing and easy expansion, and are widely used in fields such as cloud computing and big data.

[0003] In distributed storage, read / write efficiency is a key pain point. If write operations are performed directly onto the storage medium, random writes are prone to occur. Due to the mechanical characteristics of disks, random write speeds are slow, significantly increasing disk seek time and severely impacting storage performance. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, storage medium, and program product for writing data to solve the problem of random writes dragging down storage performance.

[0005] This application provides a method for writing data, including: The used storage space of multiple sub-log areas is monitored separately. The sub-log areas are used to record operation logs of write requests. If it is determined that the amount of used storage space in the first sub-log area is greater than or equal to a first preset threshold, or if the log length of the first operation log of the target write request is greater than the target remaining space in the first sub-log area, a first operation table corresponding to the first sub-log area is determined from the operation tables corresponding to the multiple sub-log areas respectively. The first sub-log area is any one of the multiple sub-log areas, the target write request includes first block identification information, the first sub-data area to which the data block corresponding to the first block identification information belongs corresponds to the first sub-log area, and the first operation table includes at least one operation information, which includes address information and data content. Sort the operation information in the first operation table according to the address information to obtain a sort of at least one operation information; Based on the order of at least one operation message, the data content in at least one operation message is sequentially flushed to the storage device.

[0006] This application also provides a data writing device, comprising: The monitoring module is used to monitor the used storage space of multiple sub-log areas, where the sub-log areas are used to record operation logs of write requests. The determination module is used to determine the first operation table corresponding to the first sub-log area from the operation tables corresponding to the multiple sub-log areas when the used storage space of the first sub-log area is greater than or equal to a first preset threshold, or the log length of the first operation log of the target write request is greater than the target remaining space of the first sub-log area. The first sub-log area is any one of the multiple sub-log areas, the target write request includes first block identification information, the first sub-data area to which the data block corresponding to the first block identification information belongs corresponds to the first sub-log area, and the first operation table includes at least one operation information, which includes address information and data content. The sorting module is used to sort the operation information in the first operation table according to the address information to obtain a sorted list of at least one operation information. The flush module is used to flush the data content of at least one operation message to the storage device in sequence according to the sorting of at least one operation message.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described data writing methods when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data writing methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data writing methods.

[0010] This application eliminates the need to directly write to the storage device upon receiving each write request. Instead, it first records logs and operation information, and then, when trigger conditions are met, performs data flushing operations from address to address. During this process, data is written sequentially, avoiding the impact of random writes on storage performance. Furthermore, in the event of an unexpected power outage, data is not lost because it is already stored in the storage device's log area. Moreover, since the sub-log area is contiguous storage space, the logs are written sequentially in the order of received write requests, eliminating the need for random writes and reducing their impact on storage performance. Based on this, this solution is adaptable to high-concurrency scenarios such as cloud computing, stably handling a large number of requests and reducing hardware expansion costs. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This application provides a schematic diagram of the architecture of a distributed storage system. Figure 2 A flowchart illustrating a data writing method provided in an embodiment of this application; Figure 3 This application provides a schematic diagram of the partitioning of a storage device. Figure 4 A schematic diagram illustrating a write request processing flow provided in an embodiment of this application; Figure 5 A schematic diagram illustrating a read request processing flow provided in an embodiment of this application; Figure 6 This is a schematic flowchart of a data writing device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0014] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0015] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0016] The data writing method provided in this application can be implemented by a distributed storage system, such as... Figure 1As shown, a distributed storage system may include at least one storage node, and each storage node may be a server. A storage node may include a central processing unit (CPU), memory, and storage devices. Storage devices may be disks.

[0017] The central processing unit can include multiple cores, which include different types of threads such as business threads, business I / O threads, and refresh threads.

[0018] The business thread is primarily used to run the business layer and the resource scheduling layer. The business layer can be a virtual machine, database, file system client, etc., used to initiate requests, such as read and write requests. The resource scheduling layer is used to determine the data area corresponding to the request based on the block identifier information carried in the request, and forward the request to the core processing bound to that data area.

[0019] Business I / O threads can be used to generate corresponding operation logs based on write requests, write the operation logs to the corresponding sub-log areas, update the operation table, and send write and read responses back to the business layer. There can be multiple business I / O threads; for example, one business I / O thread can be used to perform related read and write operations in the data area it is bound to.

[0020] The data flushing thread can be used to sort the operation information in the operation table, and then flush the data in the operation table to the storage device according to the sorting. In this way, the data flushing process can run in the background, and the business I / O thread can continue to perform related processing. Separating the flushing thread from the business I / O thread can avoid resource contention.

[0021] Memory can be used to store operation tables and log index tables.

[0022] Storage devices can be used to store logs and data.

[0023] Embodiments of this application provide a data writing method, which can be executed by the aforementioned central processing unit, such as... Figure 2 As shown, the specific processing steps of the data writing method may include: Step S201: Monitor the amount of used storage space in each of the multiple sub-log areas.

[0024] Specifically, the storage device can be configured with multiple sub-log areas. These sub-log areas can be used to record operation logs for write requests. Furthermore, the metadata of each sub-log area can include a used space field to record the amount of storage space used. The central processing unit (CPU) can monitor the used space field of each sub-log area to determine if there is sufficient remaining storage space. If it determines that a sub-log area has insufficient remaining storage space, it can perform a data flush operation on the corresponding operation table for that sub-log area.

[0025] Step S202: If it is determined that the amount of used storage space in the first sub-log area is greater than or equal to the first preset threshold, or if the log length of the first operation log of the target write request is greater than the target remaining space in the first sub-log area, determine the first operation table corresponding to the first sub-log area from the operation tables corresponding to the multiple sub-log areas respectively.

[0026] The first sub-log area can be any one of multiple sub-log areas. The first preset threshold can be the size of the first sub-log area.

[0027] The target write request may include first address information and first data content. The first address information may include first block identification information, first starting offset (which may be the starting offset within the first data block corresponding to the first block identification information), and first data length of the first data content. The first sub-data area corresponds to the first sub-log area, and the first sub-data area is the sub-data area to which the data block corresponding to the first block identification information belongs.

[0028] The first operation table can include at least one operation message. The operation message can include address information and data content. Address information can include block identifier information, starting offset, data length, etc. For example, the first operation table can be a hash table, which can use a chained hash structure to resolve hash collisions. The key of the hash table is the target identifier information of the data block, and the value is a triple structure, including an offset table, a length table, and a buffer list. The offset table can be used to store the starting offset of the data content within the data block, the length table can be used to store the data length, and the buffer list can be used to store the data content.

[0029] Specifically, in this scheme, the sub-log area is located in the storage device, while the operation table is stored in memory. The size of a sub-log area is the same as the size of its corresponding operation table, and both are used to record operation information related to write requests. To avoid excessive memory and storage space consumption due to prolonged periods without performing flush operations, this scheme uses insufficient remaining storage space in the sub-log area as the condition for triggering the data flush process.

[0030] Taking the first sub-log area as an example, when the amount of used storage space in the first sub-log area is greater than or equal to the first preset threshold, it indicates that the remaining storage space in the first sub-log area is insufficient. At this time, the first operation table bound to the first sub-log area can be determined first, and the data flushing operation can be performed based on the first operation table, that is, the data content in the first operation table is written to the storage device.

[0031] Alternatively, this solution can set another trigger condition: upon receiving a write request, if it is determined that the corresponding sub-log area cannot store the operation log generated based on the write request, a data flushing operation is performed. Specifically, after receiving the target write request and generating the first operation log based on the target write request, it is necessary to first determine whether the remaining storage space in the first sub-log area is sufficient. If it is determined that it is insufficient, the first operation table bound to the first sub-log area can be determined first, and then the data content of the first operation table can be flushed to the storage device.

[0032] Optionally, after determining the first operation table, the first operation table can be set to read-only mode, allowing only read operations and prohibiting modification of the first operation table.

[0033] Step S203: Sort the operation information in the first operation table according to the address information to obtain the sorting of at least one operation information.

[0034] Specifically, during the brushing process, all block identification information in the first operation table can be sorted from smallest to largest according to the starting offset.

[0035] Step S204: According to the sorting of at least one operation message, the data content in at least one operation message is sequentially flushed to the storage device.

[0036] Specifically, each block identifier can be traversed in sorted order. Upon encountering a block identifier, the disk offset corresponding to that block identifier can be retrieved. Then, based on the disk offset and the starting offset of the block identifier, the data corresponding to that block identifier is flushed to the corresponding data area in the storage device.

[0037] The data writing method in this application does not require directly writing to the storage device every time a write request is received. Instead, it first records logs and operation information, and then, according to preset trigger conditions, performs data flushing operations from front to back based on address information when the trigger conditions are met. During this process, data is written sequentially, avoiding the impact of random writes on storage performance. Furthermore, in the event of an unexpected power outage, since the data is already stored in the log area of ​​the storage device, no data loss will occur. Moreover, because the sub-log area is a contiguous storage space, the logs are written sequentially in the order of received write requests, eliminating the need for random writes and reducing the impact of random writes on storage performance. Based on this, this solution can be adapted to high-concurrency scenarios such as cloud computing, stably handling a large number of requests and reducing hardware expansion costs.

[0038] In some optional implementations, when the distributed storage system starts up, each storage device in the distributed storage system can be divided into a log area and a data area. Accordingly, before step S201 above, the central processing unit can also perform the following specific operations: Step 1: Obtain partition configuration information.

[0039] Step 2: Based on the partition configuration information, divide the storage space of the storage device into a log area and a data area.

[0040] Step 3: Divide the log area into multiple sub-log areas.

[0041] Step four: Divide the data area into multiple sub-data areas.

[0042] The partition configuration information can be the ratio between the log area and the data area, where the log area ratio is smaller than the data area ratio. Alternatively, it can include the sizes of the log area and the data area, where the log area size can be smaller than the data area size. The first preset unit can be 1GB. When partitioning the log space, it can be forced to align with the sector size (e.g., 4KB) to avoid performance loss caused by cross-sector writes. The second preset unit can be 4MB.

[0043] Specifically, the pre-set partition configuration information is first obtained. Then, the storage space of each storage device is partitioned according to this configuration information to obtain a log area and a data area that match the partition configuration. For example, for each available disk, it can be partitioned according to the default ratio of "log area: data area" = "m:n", allocating 10GB of log area and 90GB of data area to a 100GB disk. Disk partitioning can be done as follows: Figure 3 As shown.

[0044] The log area can be divided into multiple sub-log areas of equal size according to a first preset unit. That is, the size of the sub-log area is the first preset unit. The divided sub-log areas form a log pool, and each sub-log area corresponds to a starting offset.

[0045] The data area can be divided into multiple data blocks according to a second preset unit, meaning the size of each data block is the second preset unit. Each data block can correspond to a starting offset. Then, these multiple data blocks are grouped to obtain multiple sub-data areas. For example, the sub-data areas can be divided as follows: Obtain the number of idle cores on the service device. Divide multiple data blocks into multiple sub-data regions, equal to the number of idle cores. For example, if the CPU has 16 idle cores, the data blocks in the data region can be divided into 16 sub-data regions. During the division process, the data blocks in each sub-data region should be as evenly distributed as possible.

[0046] In this way, partitioning avoids excessive logs taking up storage space for data content. Furthermore, partitioning simplifies management; for example, when recording operation logs, they only need to be recorded in the bound sub-log area, and when recording operation tables, they only need to be recorded in the bound operation tables, eliminating the need for complex addressing operations.

[0047] In some optional implementations, to facilitate the management of sub-data areas, after the partitioning operation is completed, a data area identifier can be assigned to each sub-data area, and an initial block identifier can be assigned to each data block within that sub-data area. Then, for each data block, a block identifier can be generated based on the data area identifier of its sub-data area and the initial block identifier, for example, Data-01-0001, where 01 represents the data area identifier and 0001 represents the initial block identifier. Furthermore, for any sub-data partition, data block distribution information can be generated based on the block identifiers of its constituent data blocks. For example, for partition 1, its data block distribution information consists of data blocks Data-01-0001 to Data-01-0100.

[0048] In some optional implementations, after the partitioning operation is completed, identification information can be assigned to each sub-log area. Accordingly, the central processing unit can also perform the following specific steps: Assign log area identifiers to multiple sub-log areas. Generate a log index table based on the log area identifiers, starting offsets, and preset initial states of the multiple sub-log areas.

[0049] Specifically, to facilitate subsequent log recording, log area identification information can be assigned to each sub-log area. The log area identification information can be determined based on the identification information of the storage device to which the sub-log area belongs and the sequence number of the sub-log area in the storage device (also known as the block number of the log block). The format of the log area identification information can be "WAL-disk number-block number", for example, WAL-01-0001 can represent the first sub-log area in disk 01.

[0050] Furthermore, to facilitate subsequent management of each sub-log area, a corresponding state can be set for each sub-log area. For example, the state can be idle, occupied, or pending reclamation. The default initial state is idle. After binding a sub-log area to a sub-data area, the state can be changed to occupied. After determining that the sub-log area meets the conditions for triggering the data flushing process, the state can be changed to pending reclamation. After the data flushing process is completed, the state can be changed back to idle. In the idle state, the sub-log area can be allocated; in the occupied state, the sub-log area cannot be allocated but can be written to the operation log; in the pending reclamation state, the sub-log area cannot be written to the operation log.

[0051] Finally, for each sub-log area, a status data can be generated based on the sub-log area identifier information, starting offset, and preset initial state. The status data of all sub-log areas constitute the log index table, which is also the index table of the log pool. The log index table can be stored in memory, storage devices, etc.

[0052] In this way, the central processing unit can quickly allocate and bind sub-log areas through the log index table.

[0053] In some optional implementations, the number of sub-data areas can be less than or equal to the number of CPU cores, allowing each core to process data in parallel, thus improving read and write efficiency. Accordingly, after completing the sub-data area partitioning operation, the CPU can also perform the following specific operations: Step 1: Bind a core to a sub-data area.

[0054] Step 2: Create corresponding operation tables for each of the multiple sub-data areas.

[0055] Step 3: Bind the corresponding operation table and sub-log area to the multiple sub-data areas respectively.

[0056] Specifically, in step one, an equal number of cores can be allocated from the CPU's core pool based on the number of sub-data partitions. Each core is then bound to a sub-data partition; for example, partition 1 is bound to CPU0, and partition 2 is bound to CPU1. Furthermore, affinity can be set using kernel tools (e.g., taskset) to prevent other processes from occupying the bound cores, thus reducing the impact of context switching on I / O performance.

[0057] In step two, for any sub-data partition, a fixed-size (e.g., 1G) operation table can be created in memory.

[0058] In step three, for each sub-data area, the corresponding operation table can be bound to it, and a sub-log area can be allocated and bound to it. For example, the mapping relationship between the data area identifier information of the sub-data area, the identifier information of the operation table or its storage location in memory, and the log area identifier information of the sub-log area can be recorded.

[0059] Optionally, for any sub-data area, after receiving the first write request corresponding to it, a sub-log area and an operation table can be bound to that sub-data area.

[0060] In this way, by dividing the data into sub-data areas and sub-log areas, and creating an operation table, operations on one sub-data area are synchronized to the sub-log area and the operation table. Read and write operations between multiple sub-data areas can be performed in parallel, resulting in high read and write efficiency. Furthermore, for multiple write requests to the same sub-data area, the log is recorded in a sequential manner, which can improve write data efficiency.

[0061] In some optional implementations, whenever a write request is received, the sub-log area and operation table can be updated accordingly based on the received write request. Specifically, taking the target write request as an example, the central processing unit can perform the following operations: Step 1: Obtain the target write request.

[0062] Step 2: Based on the first identification information and the data block distribution information of multiple sub-data areas, determine the first sub-data area among the multiple sub-data areas.

[0063] Step 3: Based on the target write request, generate the first operation log corresponding to the target write request, and determine the log length of the first operation log.

[0064] Step 4: Obtain the target remaining space of the first sub-log area corresponding to the first sub-data area.

[0065] Step 5: If the remaining space of the target is greater than the log length, write the first operation log to the first sub-log area.

[0066] Step 6: Update the first operation table based on the first block identifier information, the first starting offset, the first data length, and the first data content.

[0067] Step 7: Feedback the write response corresponding to the target write request.

[0068] Specifically, such as Figure 4 As shown, in step one, after the business layer issues the target write request, the resource scheduling layer can obtain the target write request.

[0069] In step two, the resource scheduling layer can first parse the target write request to obtain the first block identification information, the first address information, and the first data content. Based on the first block identification information and the data block distribution information of multiple pre-constructed sub-data areas, the resource scheduling layer can determine the first data block distribution information to which the first block identification information belongs, and determine the sub-data area corresponding to the first data block distribution information as the first sub-data area.

[0070] In step three, a first operation log can be generated based on the information carried in the target write request. This can be achieved by converting the information into a fixed-format serial binary string. Then, the length of this binary string is determined to obtain the log length. For example, the structure of the first operation log can be as shown in Table 1.

[0071] Table 1. Structure of the Operation Log

[0072] The checksum can be obtained by performing a cyclic redundancy check (CRC) on other contents of the operation log (including block identifier information, starting offset, length, data content, etc.).

[0073] In step four, before storing the first operation log, the first sub-log area bound to the first sub-data area can be determined. Then, the amount of used storage space in the first sub-log area can be obtained from the used space field in the metadata of the first sub-log area. Since the size of each sub-log area is a fixed first preset unit, the target remaining space can be obtained by subtracting the used space of the first sub-log area from the first preset unit.

[0074] In step five, the target remaining space can be compared with the log length. If the target remaining space is greater than the log length, it means that the storage space of the first sub-log area is sufficient. At this time, the first operation log can be directly written to the first sub-log partition in an append-only manner, and the value of the used space field of the first sub-log area can be updated, that is, the log length is added to the original value.

[0075] In step six, after successfully completing the log writing operation, the first operation table can be updated based on the first address information carried in the target write request.

[0076] In step seven, after updating the first operation table, a write response can be sent back to the business layer to indicate that the target write request has been completed.

[0077] In this way, prioritizing the writing of operation logs can prevent data loss. Furthermore, checking the first sub-log area before writing the operation log can prevent log writing failures and trigger the data flushing process in a timely manner.

[0078] In some alternative implementations, step six above continues to refer to... Figure 4 The first operation table is updated based on the first identifier information, the first starting offset, the first data length, and the first data content. This update may include: Step 1: Determine whether the first block of identification information exists in the first operation table.

[0079] Step 2: If it is determined that there is a first block of identification information in the first operation table, modify the operation data corresponding to the first block of identification information according to the first address information and the first data content.

[0080] Alternatively, in step 3, if it is determined that there is a first block of identification information in the first operation table, the operation information corresponding to the first block of identification information is modified according to the first starting offset, the first data length, and the first data content.

[0081] Specifically, we can first determine whether the first block of identification information exists in the first operation table. If it does, it means that the update is for data in the same data block. At this time, we can determine the operation information corresponding to the first block of identification information based on the first block of identification information, and extract the original starting offset, original data length, and original data content from the operation information. Further, we can compare the original starting offset and the original data length, and the first starting offset and the first data length to determine the latest starting offset and the latest data length. We can then update the original starting offset to the latest starting offset, update the original data length to the latest data length, and update the original data content based on the first data content. For example, referring to the structure of the aforementioned hash table, we can add or overwrite the corresponding starting offset in the offset table, update the corresponding length in the length table, and replace or append data in the buffer list. Assuming the original starting offset is 4, the first starting offset is 2, the original data length is 2, the first data length is 2, the latest starting offset is 2, and the latest data length is 4, we can then add the first data content before the original data content.

[0082] If not, a key-value pair can be added to the first operation table, using the first block identifier information as the key and the first starting offset, first data length, and first data content as the values. For example, the first starting offset can be written to the associated offset table, the first data length can be written to the associated length table, and the first data content can be written to the associated buffer list.

[0083] In some optional implementations, in step five above, after the comparison operation, if it is determined that the target remaining space is less than or equal to the log length, then in addition to performing the data flushing process, the central processing unit may also execute the following specific steps: Step 1: Determine the identifier information of the target log area that is in an idle state in the log index table.

[0084] Step 2: Set the status corresponding to the target log area identifier information in the log index table to occupied.

[0085] Step 3: Create the second operation table.

[0086] Step four: Bind the second sub-log area and the second operation table, which correspond to the target log area identifier information, to the first sub-data area respectively.

[0087] Step 5: Add the first operation log to the second sub-log area.

[0088] Step 6: Add the first block of identification information, the first block of address information, and the first block of data content to the second operation table.

[0089] Specifically, if the target remaining space is less than or equal to the log length, it means that the remaining storage space of the first sub-log area is insufficient. In this case, in order to complete the write data operation in a timely manner, it is possible to first determine whether there are any unoccupied sub-log areas. For example, determine whether there is any log area identification information with a status of free in the log index table.

[0090] If it exists, a log area identifier can be selected from the idle log area identifiers as the target log area identifier, and its corresponding status can be updated from idle to occupied. Additionally, a new second operation table can be created for the first sub-data area. The structure of the second operation table is similar to that of the first operation table, and will not be repeated here. Then, both the second sub-log area and the second operation table can be bound to the first sub-data area. For example, a mapping relationship can be recorded between the data area identifier of the first sub-data area, the identifier of the second operation table, or the memory storage location, and the log area identifier of the second sub-log area. Furthermore, key-value pairs can be added to the second operation table, where the key is the first block identifier, and the value is the first address information and the first data content. For example, the first starting offset can be written to the associated offset table, the first data length can be written to the associated length table, and the first data content can be written to the associated buffer list.

[0091] In this way, when the remaining storage space in the currently occupied first sub-log area is insufficient, a new second sub-log area can be allocated in a timely manner to store the first operation log, allowing for prompt completion of data write operations. Simultaneously, the data corresponding to the first sub-log area can be written and processed in parallel during the flushing process, resulting in high efficiency. Furthermore, a new second operation table is created concurrently, enabling the first sub-log area to be bound to the new second sub-log area and the second operation table, facilitating management.

[0092] In some optional implementations, after the data flushing operation is completed, the first operation table can be deleted to release the memory space occupied by the first operation table. Additionally, the operation logs in the first sub-log area can be deleted, and the status of the first sub-log area can be updated to free. For example, the status field in the status data containing the log area identifier information of the first sub-log area in the aforementioned log index table can be updated to free. This allows memory and the log area to be freed up promptly. Furthermore, after the data flushing operation is completed, the central processing unit can unbind the first sub-data area from the first sub-log area and the first operation table, for example, by deleting the relevant mapping relationships.

[0093] In some alternative implementations, if the target remaining space is determined to be less than or equal to the log length, the central processing unit may also update the state of the first sub-log area to the first state.

[0094] The first state can be used to indicate that writing logs to the first sub-log area is prohibited. The first state can be the above-mentioned pending reclamation state.

[0095] Specifically, the central processing unit (CPU) can update the status field of the status data corresponding to the first block identification information in the log index table to the first status. After being marked as the first status, whenever a write request for the first sub-data area is received, it can first determine the status of each sub-log area it is bound to. If the status is the first status, then other sub-log areas are used. For example, the first sub-data area mentioned above is bound to a first sub-log area and a second sub-log area. The first sub-log area is in a pending reclaiming state, and the second sub-log area is in a occupied state. After receiving a new write request for the first sub-data area, if the data content in the first operation table has not yet been flushed down, and the binding relationship has not yet been released, then the second sub-log area in the occupied state can be selected for use.

[0096] Alternatively, the central processing unit can immediately unbind the first sub-data area from the first sub-log area and the first operation table after determining that the remaining space of the target is less than or equal to the log length. This allows for more timely processing without having to select an occupied sub-log area from among multiple sub-log areas when a new write request is received.

[0097] In some optional implementations, referring to the foregoing embodiments, in order to improve data read and write efficiency, a corresponding core is bound to each sub-data area. Accordingly, in step two above, after determining the first sub-data area among the multiple sub-data areas based on the first block identifier information and the data block distribution information of the multiple sub-data areas, the following steps can also be performed: The target write request is scheduled to the first core bound to the first sub-data area.

[0098] Specifically, after the resource scheduling layer determines the first sub-data area, it can forward the target write request to the first core bound to the first sub-data area, and the first core will execute the subsequent steps to avoid cross-core scheduling delay.

[0099] In some alternative implementations, based on the foregoing embodiments, after the methods of writing data and data flushing processes are changed, the methods of reading data also change accordingly, such as... Figure 5 As shown, the central processing unit can process read requests as follows: Step 1: Obtain the read request.

[0100] The read request includes second address information, which includes second block identifier information, second starting offset, and second data length.

[0101] Step 2: Based on the second block identification information and the data block distribution information of multiple sub-data areas, determine the second sub-data area among the multiple sub-data areas.

[0102] Step 3: Based on the second identification information, determine whether there is candidate operation information in the second operation table that includes the second identification information.

[0103] The second operation table is the operation table corresponding to the second sub-data area, and the candidate operation information also includes the third starting offset and the third data length.

[0104] Step 4: If candidate operation information is determined, the comparison result is determined based on the second starting offset, the second data length, the third starting offset, and the third data length.

[0105] The candidate operation information includes third address information and third data content. The third address information includes second block identifier information, third starting offset, and third data length.

[0106] Step 5: Based on the comparison results, perform the read operation corresponding to the comparison results to obtain the read results.

[0107] Step six: If no candidate operation information is found, perform a read operation on the storage device according to the second address information to obtain the read result.

[0108] Step 7: Feedback on the reading results.

[0109] Specifically, in step one, after the business layer issues a read request, the resource scheduling layer can obtain the read request, parse the read request, and obtain the second block of identification information, the second starting offset, and the second data length, etc.

[0110] In step two, the resource scheduling layer can determine the second data block distribution information to which the second block identification information belongs based on the second block identification information and the data block distribution information of multiple pre-built sub-data areas, and determine the sub-data area corresponding to the second data block distribution information as the second sub-data area.

[0111] Optionally, after the second sub-data area is determined, read requests can be forwarded to the second core bound to the second sub-data area so that the second core can perform subsequent steps.

[0112] In step three, it can be first determined whether the first two blocks of identification information exist in the second operation table bound to the second sub-data area. If so, the operation information to which the second block of identification information belongs can be identified as candidate operation information. At this point, the second address information can be compared with the address information in the candidate operation information to determine the comparison result. For example, the comparison result can have the following multiple possibilities: First, the candidate operation information includes all data content that matches the second address information. For example, the third starting offset covers the second starting offset, and the third data length is greater than or equal to the second data length.

[0113] Second, the candidate operation information includes part of the data content corresponding to the second address information, and the remaining data content is in the storage device.

[0114] Third, the candidate operation information does not include the data content corresponding to the second address information.

[0115] In the first case, all data content can be read directly from the third data content of the candidate operation information based on the second address information.

[0116] In the second scenario, we can first read initial data content of the same length as the second data length from the storage device, based on the second starting offset. Then, based on the second address information, we can read a portion of the data content from the third data content of the candidate operation information. Since the read portion of the data content is the latest data content and has not yet been flushed, we can replace the data content at the corresponding position in the initial data content with the portion of the data content to ensure that the read data content is the latest data content.

[0117] In the third scenario, or if the second block identifier information is not present in the second operation table, the data content can be directly read from the storage device based on the second address information. Optionally, when reading the data content, the complete data content in the data block corresponding to the second block identifier information can be read first, and a CRC calculation can be performed on the complete data content to obtain a check value. This check value is then compared with the check value recorded in the data block. If the comparison matches, the data content corresponding to the second address information can be extracted from the complete data content.

[0118] After completing the read operation, the read result can be generated based on the read data content and fed back to the business layer.

[0119] This prioritizes reading data from memory, resulting in higher efficiency and significantly reduced read latency. Furthermore, read operations on different sub-data areas can be executed in parallel, further improving read efficiency.

[0120] In some alternative implementations, a log index table may also be stored in the storage device, and the central processing unit may accordingly perform the following operations: Step 1: After restarting, obtain the status of multiple sub-log areas in the storage device.

[0121] Step 2: Filter candidate sub-log areas that are currently occupied.

[0122] Step 3: Read at least one operation log from the candidate sub-log area.

[0123] Step 4: Based on at least one operation log, create an operation table corresponding to the candidate sub-log area.

[0124] Specifically, after a reboot, the log index table can be read from the storage device to obtain the status of multiple sub-log areas. Alternatively, the header of each sub-log area can store its log area identifier, status, starting offset, etc. After a reboot, the log index table can be quickly reconstructed by scanning the headers of each sub-log area. Then, the identifiers of candidate log areas in the occupied state can be determined from the log index table. The sub-log areas corresponding to these candidate log area identifiers are then identified as candidate sub-log areas, i.e., candidate sub-log areas in the occupied state are selected. For any candidate sub-log area, all operation logs can be read from it. Then, based on the structure of the aforementioned operation table (e.g., a hash table), the necessary content for operation information can be extracted from each operation log, creating operation information corresponding to each operation log. The operation information corresponding to all operation logs constitutes the operation table corresponding to that candidate sub-log area.

[0125] In this way, after a restart, the central processing unit can continue to perform read and write operations based on the operation logs recorded in the sub-log area and the operation information recorded in the operation table.

[0126] In some optional implementations, to ensure the accuracy of the restored operation table, the integrity of the operation logs in each candidate sub-log area can be verified before step four above. Accordingly, the central processing unit can perform the following operations: Step 1: Read the reference verification information from the last operation log in at least one operation log.

[0127] Step two: Read the fourth data content and the fourth address information from the last operation log.

[0128] Step 3: Generate target verification information based on the fourth data content and the fourth address information.

[0129] Step 4: Determine the verification result based on the target verification information and the reference verification information.

[0130] Step 5: If the verification result is used to indicate that the operation log in the candidate sub-log area is complete, then reconstruct the operation table corresponding to the candidate sub-log area based on at least one operation log.

[0131] Specifically, for the last operation log entry in each candidate sub-log area, CRC can be used to calculate the target verification information based on the block identifier, starting offset, length, and data content of the operation log. This target verification information is then compared with the reference verification information originally recorded in the operation log. If they match, the operation log in the candidate sub-log area is considered complete, and an operation table corresponding to the candidate sub-log area can be created based on at least one operation log entry. If they do not match, the verification continues for the second-to-last operation log entry until a matching operation log entry is identified as the truncation point. Operation log entries after the truncation point are considered erroneous, and the operation log entries before and after the truncation point are recovered to obtain the operation table.

[0132] In this way, the operation log is written to the sub-log area in an append-only manner. Therefore, the process can be stopped once the first corrupted log entry is found at the end, resulting in high recovery efficiency and accurate location. Erroneous data is not used, ensuring the accuracy of the operation table.

[0133] In some optional implementations, each operation information in the operation table can also record a check value of the data content, such as a check value obtained by calculating the data content using CRC. Accordingly, in the data flushing process, after each operation information flushing operation is completed, the data content recorded at the corresponding position can be read again, and the check value can be recalculated and compared with the check value in the operation table. If they match, it means that the writing is successful. If they do not match, the write operation can be re-executed until the writing is successful.

[0134] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0135] Embodiments of this application also provide a data writing device, such as... Figure 6 As shown, it includes: The monitoring module 610 is used to monitor the amount of used storage space in multiple sub-log areas, where the sub-log areas are used to record operation logs of write requests. The determining module 620 is used to determine the first operation table corresponding to the first sub-log area from the operation tables corresponding to the multiple sub-log areas when the used storage space of the first sub-log area is greater than or equal to a first preset threshold, or the log length of the first operation log of the target write request is greater than the target remaining space of the first sub-log area. The first sub-log area is any one of the multiple sub-log areas, the target write request includes first block identification information, the first sub-data area to which the data block corresponding to the first block identification information belongs corresponds to the first sub-log area, and the first operation table includes at least one operation information, which includes address information and data content. The sorting module 630 is used to sort the operation information in the first operation table according to the address information to obtain the sorting of at least one operation information. The flush module 640 is used to flush the data content of at least one operation message to the storage device in sequence according to the sorting of at least one operation message.

[0136] In some alternative implementations, the device further includes a data writing module for: Obtain the target write request, wherein the target write request includes first address information and first data content, and the first address information includes first block identification information, first starting offset and first data length; Based on the first identification information and the data block distribution information of multiple sub-data areas, the first sub-data area is determined among the multiple sub-data areas; Based on the target write request, generate the first operation log corresponding to the target write request, and determine the log length of the first operation log; Get the target remaining space of the first sub-log area corresponding to the first sub-data area; If it is determined that the remaining space of the target is greater than the log length, the first operation log is written to the first sub-log area; The first operation table is updated based on the first identifier information, the first starting offset, the first data length, and the first data content. Feedback is the write response corresponding to the target write request.

[0137] In some optional implementations, after determining the first sub-data area among multiple sub-data areas based on the first block identification information and the data block distribution information of multiple sub-data areas, the method further includes: The target write request is scheduled to the first core bound to the first sub-data area.

[0138] In some optional implementations, the first operation table is updated based on the first block identifier information, the first starting offset, the first data length, and the first data content, including: Determine whether the first block of identifier information exists in the first operation table; If it is determined that there is a first block of identification information in the first operation table, the operation information corresponding to the first block of identification information in the first operation table is modified according to the first starting offset, the first data length, and the first data content. Alternatively, if it is determined that the first block of identification information does not exist in the first operation table, the first block of identification information, the first address information, and the first data content are added to the first operation table.

[0139] In some alternative implementations, the device further includes a partitioning module for: Get partition configuration information; Based on the partition configuration information, the storage space of the storage device is divided into a log area and a data area; Divide the log area into multiple sub-log areas; The data area is divided into multiple sub-data areas.

[0140] In some alternative implementations, the partitioning module is also used for: Assign log area identification information to multiple sub-log areas; A log index table is generated based on the log area identifier information, starting offset, and preset initial state of multiple sub-log areas.

[0141] In some alternative implementations, the determining module 620 is further configured to: If the remaining space of the target is determined to be less than or equal to the log length, the target log area identifier information with an idle status is determined in the log index table; Set the status corresponding to the target log area identifier information in the log index table to occupied; Create a second operation table; Bind the second sub-log area and the second operation table, which correspond to the target log area identifier information, to the first sub-data area respectively; Add the first operation log to the second sub-log area; Add the first block of identification information, the first block of address information, and the first block of data content to the second operation table.

[0142] In some alternative implementations, the determining module 620 is further configured to: If the remaining space of the target is determined to be less than or equal to the log length, the status of the first sub-log area is updated to the first status, where the first status is used to indicate that writing logs to the first sub-log area is prohibited.

[0143] In some optional implementations, after sequentially flushing the data content of at least one operation message to the storage device according to the order of at least one operation message, the method further includes: Delete the first operation table; Delete the operation logs in the first sub-log area and update the status of the first sub-log area to idle.

[0144] In some alternative implementations, the device further includes a data reading module for: Obtain a read request, wherein the read request includes second address information, the second address information includes second block identifier information, second starting offset, and second data length; Based on the second identification information and the data block distribution information of multiple sub-data areas, the second sub-data area is determined among the multiple sub-data areas; Based on the second block of identification information, determine whether there is candidate operation information including the second block of identification information in the second operation table. The second operation table is the operation table corresponding to the second sub-data area. The candidate operation information also includes the third starting offset and the third data length. If candidate operation information is determined, the comparison result is determined based on the second starting offset, the second data length, the third starting offset, and the third data length. Based on the comparison results, perform the corresponding read operation to obtain the read results; If it is determined that there is no candidate operation information, a read operation is performed on the storage device according to the second address information to obtain the read result; Feedback on the reading results.

[0145] In some alternative embodiments, the device further includes a fault handling module for: After restarting, retrieve the status of multiple sub-log areas in the storage device; Filter candidate sub-log areas that are currently occupied; Read at least one operation log from the candidate sub-log area; Reconstruct the operation table corresponding to the candidate sub-log area based on at least one operation log.

[0146] In some optional implementations, the fault handling module is also used for: Read reference verification information from the last operation log in at least one operation log; Read the fourth data content and the fourth address information from the last operation log; Based on the fourth data content and the fourth address information, generate target verification information; The verification result is determined based on the target verification information and the reference verification information; If the verification result is determined to indicate that the operation logs in the candidate sub-log area are complete, then the operation table corresponding to the candidate sub-log area is reconstructed based on at least one operation log.

[0147] For a description of the features in the embodiment corresponding to the data writing device, please refer to the relevant description in the embodiment corresponding to the data writing method, which will not be repeated here.

[0148] Embodiments of this application also provide an electronic device, such as... Figure 7 As shown, it includes a memory 10 and a processor 20. The memory 10 stores a computer program, and the processor 20 is configured to run the computer program to perform the steps in any of the above-described data writing method embodiments.

[0149] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data writing method embodiments when it is run.

[0150] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0151] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data writing method embodiments.

[0152] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data writing method embodiments.

[0153] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (fixed logic circuit systems), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microcontroller unit (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination thereof. The aforementioned computer program (also known as a program, software, software application, App, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0154] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0155] The foregoing has provided a detailed description of a data writing method, apparatus, electronic device, storage medium, and program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A method for writing data, characterized in that, include: The used storage space of multiple sub-log areas is monitored separately, wherein the sub-log areas are used to record operation logs of write requests; If it is determined that the amount of used storage space in the first sub-log area is greater than or equal to a first preset threshold, or if the log length of the first operation log of the target write request is greater than the target remaining space in the first sub-log area, a first operation table corresponding to the first sub-log area is determined from the operation tables corresponding to the multiple sub-log areas respectively. The first sub-log area is any one of the multiple sub-log areas. The target write request includes a first block identification information. The first sub-data area to which the data block corresponding to the first block identification information belongs corresponds to the first sub-log area. The first operation table includes at least one operation information, which includes address information and data content. According to the address information, sort the operation information in the first operation table to obtain the sorting of at least one operation information; Based on the order of at least one of the operation messages, the data content in at least one of the operation messages is sequentially flushed to the storage device.

2. The data writing method according to claim 1, characterized in that, The method further includes: Obtain the target write request, wherein the target write request includes first address information and first data content, and the first address information includes first block identifier information, first starting offset and first data length; Based on the first block identification information and the data block distribution information of multiple sub-data areas, a first sub-data area is determined among the multiple sub-data areas; Based on the target write request, generate the first operation log corresponding to the target write request, and determine the log length of the first operation log; Obtain the target remaining space of the first sub-log area corresponding to the first sub-data area; If it is determined that the target remaining space is greater than the log length, the first operation log is written to the first sub-log area; The first operation table is updated based on the first block identifier information, the first starting offset, the first data length, and the first data content. Feedback is provided in the form of a write response corresponding to the target write request.

3. The data writing method according to claim 2, characterized in that, After determining the first sub-data area among the multiple sub-data areas based on the first block identification information and the data block distribution information of the multiple sub-data areas, the method further includes: The target write request is scheduled to the first core bound to the first sub-data area.

4. The data writing method according to claim 2 or 3, characterized in that, The step of updating the first operation table based on the first block identifier information, the first starting offset, the first data length, and the first data content includes: Determine whether the first block of identification information exists in the first operation table; If it is determined that the first block of identification information exists in the first operation table, the operation information corresponding to the first block of identification information in the first operation table is modified according to the first starting offset, the first data length, and the first data content. Alternatively, if it is determined that the first block of identification information does not exist in the first operation table, the first block of identification information, the first address information, and the first data content are added to the first operation table.

5. The data writing method according to claim 2 or 3, characterized in that, Before monitoring the used storage space of multiple sub-log areas, the method further includes: Get partition configuration information; Based on the partition configuration information, the storage space of the storage device is divided into a log area and a data area; The log area is divided into multiple sub-log areas; The data area is divided into multiple sub-data areas.

6. The data writing method according to claim 5, characterized in that, After dividing the log area into multiple sub-log areas, the method further includes: Assign log area identification information to multiple sub-log areas; A log index table is generated based on the log area identifier information, starting offset, and preset initial state of the multiple sub-log areas.

7. The data writing method according to claim 6, characterized in that, The method further includes: If the target remaining space is determined to be less than or equal to the log length, the target log area identifier information with an idle status is determined in the log index table. Set the status corresponding to the target log area identifier information in the log index table to occupied; Create a second operation table; The second sub-log area and the second operation table, which correspond to the target log area identification information, are respectively bound to the first sub-data area; Add the first operation log to the second sub-log area; The first block identifier information, the first address information, and the first data content are added to the second operation table.

8. The data writing method according to any one of claims 1 to 3, characterized in that, The method further includes: If it is determined that the target remaining space is less than or equal to the log length, the state of the first sub-log area is updated to a first state, wherein the first state is used to indicate that writing logs to the first sub-log area is prohibited.

9. The data writing method according to claim 8, characterized in that, After sequentially flushing the data content of at least one operation message to the storage device according to the sorting of at least one operation message, the method further includes: Delete the first operation table; Delete the operation logs in the first sub-log area and update the status of the first sub-log area to idle.

10. The data writing method according to any one of claims 1 to 3, characterized in that, The method further includes: Obtain a read request, wherein the read request includes second address information, the second address information including second block identifier information, second starting offset, and second data length; Based on the second block identification information and the data block distribution information of the multiple sub-data areas, the second sub-data area is determined among the multiple sub-data areas; Based on the second block of identification information, determine whether there is candidate operation information including the second block of identification information in the second operation table, wherein the second operation table is the operation table corresponding to the second sub-data area, and the candidate operation information also includes a third starting offset and a third data length; If the candidate operation information is determined to exist, the comparison result is determined based on the second starting offset, the second data length, the third starting offset, and the third data length. Based on the comparison result, perform the read operation corresponding to the comparison result to obtain the read result; If it is determined that no candidate operation information exists, a read operation is performed on the storage device according to the second address information to obtain a read result; The reading results will be displayed in the feedback.

11. The data writing method according to any one of claims 1 to 3, characterized in that, The method further includes: After restarting, obtain the status of the multiple sub-log areas in the storage device; Filter candidate sub-log areas whose status is occupied; Read at least one operation log from the candidate sub-log area; Based on at least one of the operation logs, reconstruct the operation table corresponding to the candidate sub-log area.

12. The data writing method according to claim 11, characterized in that, Before reconstructing the operation table corresponding to the candidate sub-log area based on at least one of the operation logs, the method further includes: Read reference verification information from the last operation log of at least one of the operation logs; Read the fourth data content and the fourth address information from the last operation log entry; Target verification information is generated based on the fourth data content and the fourth address information; The verification result is determined based on the target verification information and the reference verification information; If the verification result is determined to indicate that the operation log in the candidate sub-log area is complete, the operation table corresponding to the candidate sub-log area is reconstructed based on at least one operation log.

13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data writing method as described in any one of claims 1 to 12 when executing the computer program.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data writing method as described in any one of claims 1 to 12.

15. A computer program product, characterized in that, The computer program product includes a computer program, wherein when the computer program is executed by a processor, it implements the steps of the data writing method as described in any one of claims 1 to 12.

Images