Data writing methods, devices, electronic equipment and media

Abstract

This specification provides a data writing method, apparatus, electronic device, and medium. The method includes: receiving a first write request for first data; allocating a preset data space; updating metadata according to information in the preset data space; writing the first data into the preset data space; receiving a second write request for second data; and writing the second data into the preset data space, wherein the preset data space is greater than or equal to the sum of the data amounts of the first data and the second data.

Description

Technical Field

[0001] This specification relates to the field of big data technology, and in particular to data writing methods, devices, electronic devices and media. Background Technology

[0002] In virtualization and cloud computing environments, file systems (such as OCFS2) are widely used to provide shared storage for virtual machines across multiple hosts. When a virtual machine writes data, the file system needs to dynamically allocate disk space and update the corresponding metadata (such as inode and extent information). The traditional space allocation strategy is "on-demand allocation," meaning that only physical space matching the size of the data being written is allocated each time. This strategy leads to frequent metadata update operations. Since metadata needs to be consistent across all nodes in the cluster, each update may trigger cross-node synchronization and I / O (input / output) operations, resulting in severe "metadata write amplification." This not only consumes valuable storage I / O bandwidth but also occupies a significant amount of CPU (central processing unit) resources, ultimately leading to a significant degrade in the virtual machine's business I / O performance, especially in scenarios involving a large number of small I / O writes. Summary of the Invention

[0003] To overcome the problems existing in related technologies, this specification provides data writing methods, devices, electronic devices, and media.

[0004] According to a first aspect of the embodiments of this specification, a data writing method is provided, the method comprising: receiving a first write request for first data; allocating a preset data space; updating metadata according to information of the preset data space; writing the first data into the preset data space; receiving a second write request for second data; and writing the second data into the preset data space, wherein the preset data space is greater than or equal to the sum of the data amounts of the first data and the second data.

[0005] According to a second aspect of the embodiments of this specification, a data writing apparatus is provided, comprising: a first request receiving module for receiving a first write request for first data; a first allocation module for allocating a preset data space; an update module for updating metadata according to information of the preset data space; a first write module for writing the first data into the preset data space; a second request receiving module for receiving a second write request for second data; and a second write module for writing the second data into the preset data space, wherein the preset data space is greater than or equal to the sum of the data amounts of the first data and the second data.

[0006] According to a third aspect of the embodiments of this specification, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to execute the data writing method of the first aspect described above or any corresponding embodiment thereof.

[0007] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a computer to perform the data writing method of the first aspect or any corresponding embodiment described above.

[0008] The technical solutions provided in the embodiments of this specification may include the following beneficial effects: In the embodiments described in this specification, by pre-allocating a preset data space, data can be directly written into the preset data space without the need for metadata updates, thus avoiding "metadata write amplification" and improving business I / O performance.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this specification and, together with the description, serve to explain the principles of this specification.

[0011] Figure 1 This is a schematic diagram of a system architecture illustrated in this specification according to an exemplary embodiment.

[0012] Figure 2 This is a flowchart illustrating a data writing method according to an exemplary embodiment of this specification.

[0013] Figure 3A This is a schematic diagram of the structure of an Inode shown in this specification according to another exemplary embodiment.

[0014] Figure 3B This is a schematic diagram illustrating a data writing method according to another exemplary embodiment of this specification.

[0015] Figure 4 This is a hardware structure diagram of a computer device containing the data writing device in an embodiment of this specification.

[0016] Figure 5 This is a block diagram illustrating a data writing apparatus according to an exemplary embodiment of this specification. Detailed Implementation

[0017] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this specification as detailed in the appended claims.

[0018] The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” as used in this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0019] It should be understood that although the terms first, second, third, etc., may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this specification, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0020] The embodiments described in this specification will now be described in detail.

[0021] The following combination Figure 1 The file system to which the data writing method and apparatus can be applied in the embodiments of this specification will be described. It should be noted that... Figure 1 The examples shown are merely file systems that can be applied to the embodiments of this specification, in order to help those skilled in the art understand the technical content of this specification, but do not mean that the embodiments of this specification cannot be used in other devices, systems, environments or scenarios.

[0022] Figure 1 This is a schematic diagram of a file system illustrated in this specification according to an exemplary embodiment.

[0023] like Figure 1As shown, this file system may include, for example, compute nodes and a storage tier (SAN storage / LUN). There can be one or more compute nodes. Compute nodes may include, for example, a KVM (Kernel-based Virtual Machine) host. A KVM host may include QEMU and OCFS2. QEMU is the core component of KVM. It is responsible for simulating computer hardware (such as CPU, network card, hard drive, etc.) and running virtual machine instances on it. OCFS2 is a clustered file system. OCFS2 allows multiple hosts to simultaneously mount and read / write the same disk partition while ensuring data consistency.

[0024] The storage tier can include a SAN (Storage Area Network). A SAN is a high-speed, dedicated network used to connect servers and storage devices. A SAN can be divided into one or more logical units, each identified by a LUN (Logical Unit Number).

[0025] The data writing method provided in the embodiments of this specification will be described in detail below. For example... Figure 2 As shown, Figure 2 This is a flowchart illustrating a data writing method according to an exemplary embodiment of this specification. The method can be applied to a file system. The data writing method provided in the embodiments of this specification may include the following steps.

[0026] In step 210, the first write request for the first data is received.

[0027] According to embodiments of this specification, a first write request may be used, for example, to request that first data be written to a target file in a file system.

[0028] In step 220, a preset data space is allocated.

[0029] According to embodiments of this specification, a preset data space is located in a storage device (e.g., a storage array). When allocating the preset data space, a zero-write operation needs to be performed on the preset data space. Optionally, the file system can send a zero-write instruction to the storage device to instruct the storage device to perform a zero-write operation on the preset data space. By completely delegating the zeroing operation of the preset data space to the backend storage device, the hardware acceleration capabilities of the storage device can be utilized to achieve fast and efficient zero-filling.

[0030] Optionally, the file system can obtain the execution result of the write zero operation of the storage device. If the write zero operation of the storage device fails, it can switch to performing the write zero operation locally to ensure that the preset data space can be allocated normally.

[0031] In step 230, update the metadata according to the information in the preset data space.

[0032] According to embodiments of this specification, metadata is data that describes data, such as file system inode and extent information, and is used to manage the physical location and attributes of the data. An inode is an index node used to store file metadata, and an extent represents a contiguous range of physical blocks, including fields such as the starting logical block number, the number of contiguous blocks, and the starting physical block number.

[0033] In step 240, the first data is written into the preset data space.

[0034] In step 250, a second write request for the second data is received.

[0035] According to embodiments of this specification, a second write request may be used, for example, to request that second data be written to a target file in the file system.

[0036] In step 260, the second data is written into the preset data space.

[0037] According to embodiments of this specification, the size of the preset data space can be set as needed. When allocating the preset data space, a larger space can be allocated so that the preset data space can accommodate multiple data items, i.e., the preset data space is greater than or equal to the sum of the amounts of the first and second data items.

[0038] According to the embodiments of this specification, by allocating a preset data space, in subsequent write operations, the file system directly locates the physical address within the pre-allocated preset data space and writes it, without needing to modify the metadata again. Therefore, no additional metadata I / O (input / output) is generated. According to the embodiments of this specification, by pre-allocating the preset data space, data can be directly written into the preset data space without requiring metadata updates, avoiding "metadata write amplification," improving effective I / O bandwidth utilization, significantly reducing write latency, and improving business I / O performance. The performance improvement is particularly significant in scenarios with small I / O writes.

[0039] Optionally, sending a write-zero instruction to the storage device may include, for example, creating a block I / O request structure (BIO); setting the operation type of the block I / O request structure to a write-zero operation type, such as WRITE_ZEROES; submitting the block I / O request structure for processing by the block device layer; and sending a write-zero instruction to the storage device. The write-zero instruction can be implemented using the WRITE SAME command in the SCSI (Small Computer System Interface) protocol.

[0040] Optionally, when submitting a block I / O request structure, the processing status of the block I / O request structure can also be obtained; if the processing status indicates processing failure, it is determined that the storage device failed to perform a write-zero operation. In the event that the storage device fails to perform a write-zero operation, it can be downgraded to a local write-zero operation.

[0041] Optionally, the file system can also monitor the remaining space of the preset data space. If the remaining space of the preset data space is less than the space threshold, a new preset data space will be allocated. The allocation of a new preset data space can be found above and will not be repeated here.

[0042] The data writing method is described below with reference to another embodiment. This embodiment describes a data writing process based on the foregoing embodiments, including the following steps: Step 1: Trigger data space pre-allocation.

[0043] According to the embodiments of this specification, when a file system (such as OCFS2) detects that data needs to be written to a file, but there is no allocated preset data space or the remaining space of the allocated preset data space is insufficient, a data space pre-allocation process can be triggered.

[0044] Step 2: Pre-allocate preset data space.

[0045] According to embodiments of this specification, the file system can pre-allocate a large contiguous block of space, i.e., a preset data space, for the file at once, and update the inode and extent metadata to record the allocation of this large block of space. Although the I / O volume of this metadata update is large, it occurs very infrequently. For example, in this embodiment, 120MB of space can be allocated as the preset data space.

[0046] Step 3: Offload the write zero operation of the preset data space to the storage device for execution.

[0047] According to embodiments in this specification, the file system can utilize the SCSI WriteSame instruction to completely delegate the zeroing operation of this preset data space to the backend storage device (such as a storage array). The WriteSame instruction is highly efficient, allowing the storage device to quickly complete the internal zeroing.

[0048] Step 4: Write subsequent data into the preset data space. According to the embodiments of this specification, when writing data to the file later, the file system can directly locate the physical address and write it within the pre-allocated preset data space without modifying the Inode or Extend metadata again, thus avoiding additional metadata I / O and achieving higher write performance.

[0049] According to the embodiments of this specification, when the pre-allocated preset data space is about to be exhausted, steps one to four above can be repeated to allocate the next preset data space for the file.

[0050] like Figure 3A The diagram shown illustrates the structure of an inode (index node) according to an exemplary embodiment. Taking the OCFS2 (Oracle Cluster File System 2) file system as an example, the structure of an inode in the file system may include, for example: ocfs2_di node: This is an inode structure unique to OCFS2, used to store basic information about the file; Metadata block: This includes Extent metadata, which is "range metadata". In a file system, to efficiently manage large files, the physical address of each data block is usually not recorded; instead, a contiguous block range (Extent) is recorded. This is where the location information of these data blocks is stored.

[0051] Data cluster: A cluster is the smallest unit of storage space allocated by a file system. For example, in this embodiment, the cluster size is 1MB. The "1MB" within the dashed box represents a new cluster just allocated for newly written data.

[0052] The relevant technology allocates a corresponding number of clusters based on the data size each time data is written. Each cluster allocation generates a corresponding metadata update operation. As a result, a large amount of metadata I / O is generated, causing "metadata write amplification" and impacting business I / O performance.

[0053] like Figure 3BThe diagram shown is a schematic representation of a data writing method according to an exemplary embodiment.

[0054] In this embodiment, by pre-allocating 120MB of data space, subsequent data can be directly written into this 120MB space without requiring metadata updates, thus avoiding "metadata write amplification." This improves effective I / O bandwidth utilization, significantly reduces write latency, and enhances business I / O performance. The performance improvement is particularly significant in scenarios with small I / O writes. Furthermore, when allocating data space, zero-write operations can be offloaded to the storage device, enabling Fast Write Zero.

[0055] Corresponding to the embodiments of the aforementioned methods, this specification also provides embodiments of a data writing device and the terminal to which it is applied.

[0056] The embodiments of the data writing device described in this specification can be applied to computer devices, such as servers or terminal devices. The device embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor reading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of a computer device containing the data writing device according to an embodiment of this specification. Except for... Figure 4 In addition to the processor 410, memory 430, network interface 420, and non-volatile memory 440 shown, the server or electronic device where the device 431 is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0057] like Figure 5 As shown, Figure 5 This is a block diagram illustrating a data writing apparatus according to an exemplary embodiment of the present specification, the apparatus comprising: The first request receiving module 510 is used to receive the first write request of the first data; The first allocation module 520 is used to allocate preset data space; Update module 530 is used to update metadata based on information from a preset data space; The first writing module 540 is used to write the first data into a preset data space; The second request receiving module 550 is used to receive the second write request of the second data; The second writing module 560 is used to write the second data into a preset data space, where the preset data space is greater than or equal to the sum of the amounts of the first data and the second data.

[0058] Optionally, the preset data space is located in the storage device, and the allocation module may include: The write-zero instruction submodule is used to send write-zero instructions to the storage device to instruct the storage device to perform a write-zero operation for a preset data space.

[0059] Optionally, the device may further include: The local write-zero module is used to perform a write-zero operation locally if the storage device fails to perform a write-zero operation.

[0060] Optionally, the device may further include: The second allocation module is used to allocate a new preset data space if the remaining space of the preset data space is less than the space threshold.

[0061] Optionally, the write-zero indicator submodule may include: Create a unit for creating a block I / O request structure; The setting unit is used to set the operation type of the block I / O request structure to the write zero operation type; The submission unit is used to submit a block I / O request structure so that the block device layer can process the block I / O request structure and send a write zero instruction to the storage device. The write zero instruction is implemented through the WRITE SAME command in the SCSI protocol.

[0062] Optionally, the device may further include: The status acquisition module is used to acquire the processing status of the block I / O request structure; The result determination module is used to determine that the storage device failed to perform a write-zero operation if the processing status indicates that the processing failed.

[0063] Accordingly, this specification also provides an electronic device, which includes a processor; a memory for storing processor-executable instructions; wherein the processor is configured to: receive a first write request for first data; allocate a preset data space; update metadata according to information in the preset data space; write the first data into the preset data space; receive a second write request for second data; and write the second data into the preset data space, wherein the preset data space is greater than or equal to the sum of the data amounts of the first data and the second data.

[0064] According to the embodiments in this specification, by pre-allocating a preset data space, data can be directly written into the preset data space without requiring metadata updates, avoiding "metadata write amplification," effectively improving I / O bandwidth utilization, significantly reducing write latency, and improving business I / O performance. The performance improvement is particularly significant in scenarios with small I / O writes.

[0065] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0066] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the solution in this specification according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0067] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0068] Other embodiments of this specification will readily occur to those skilled in the art upon consideration of the specification and practice of the invention claimed herein. This specification is intended to cover any variations, uses, or adaptations that follow the general principles of this specification and include common knowledge or customary techniques in the art not claimed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this specification are indicated by the following claims.

[0069] It should be understood that this specification is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this specification is limited only by the appended claims.

[0070] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A data writing method, applied to a file system, characterized in that, The method includes: Receive the first write request for the first data; Allocate preset data space; Update the metadata based on the information in the preset data space; Write the first data into the preset data space; Receive the second write request for the second data; The second data is written into the preset data space, where the preset data space is greater than or equal to the sum of the data amounts of the first data and the second data.

2. The method according to claim 1, characterized in that, The preset data space is located in the storage device, and the allocation of the preset data space includes: Send a write-zero instruction to the storage device to instruct the storage device to perform a write-zero operation for the preset data space.

3. The method according to claim 2, characterized in that, The method further includes: If the storage device fails to perform the write-zero operation, the write-zero operation is performed locally.

4. The method according to claim 1, characterized in that, The method further includes: If the remaining space in the preset data space is less than the space threshold, then a new preset data space is allocated.

5. The method according to claim 2, characterized in that, Sending a write-zero instruction to the storage device includes: Create a block I / O request structure; Set the operation type of the block I / O request structure to write zero operation type; The block I / O request structure is submitted so that the block device layer can process the block I / O request structure and send a write zero instruction to the storage device. The write zero instruction is implemented through the WRITE SAME command in the SCSI protocol.

6. The method according to claim 5, characterized in that, Sending a write-zero instruction to the storage device includes: Create a block I / O request structure; Set the operation type of the block I / O request structure to write zero operation type; Submit the block I / O request structure to send a write-zero instruction to the storage device, the write-zero instruction being implemented via the WRITE SAME command in the SCSI protocol; The method further includes: Obtain the processing status of the block I / O request structure; If the processing status indicates processing failure, it is determined that the storage device failed to execute the write-zero operation.

7. A data writing device, characterized in that, The device includes: The first request receiving module is used to receive the first write request of the first data; The first allocation module is used to allocate preset data space; The update module is used to update metadata based on the information in the preset data space; The first writing module is used to write the first data into the preset data space; The second request receiving module is used to receive the second write request for the second data. The second writing module is used to write the second data into the preset data space, wherein the preset data space is greater than or equal to the sum of the data amounts of the first data and the second data.

8. The apparatus according to claim 7, characterized in that, The preset data space is located in the storage device, and the allocation module includes: The write-zero instruction submodule is used to send a write-zero instruction to the storage device to instruct the storage device to perform a write-zero operation for the preset data space.

9. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store processor-executable instructions; The processor is configured to perform the data writing method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to perform the data writing method according to any one of claims 1 to 6.

Images