A data writing method, device and electronic equipment

Abstract

Embodiments of the present application provide a data writing method and device and electronic equipment, and relate to the technical field of data storage. The specific implementation scheme is as follows: when a target storage block meets a data playback condition, determining target data to be played back in the target storage block; wherein the target storage block is a first type of storage block in which data is written in an append write mode; determining whether the target data is a full stripe; if not, performing stripe alignment processing on the target data in the target storage block to obtain aligned data; and writing the aligned data into a specified storage block corresponding to a storage location of the target data, the specified storage block being a second type of data block in which data is written in a random write mode. It can be seen that, by the present scheme, the problem of write operation delay can be solved while ensuring data playback.

Description

Technical Field

[0001] This invention relates to the field of data storage technology, and in particular to a data writing method, apparatus, and electronic device. Background Technology

[0002] Because append-only writes are highly efficient, when a backend device receives a write command from a frontend device, it appends the data to be written to the LogBlock to improve efficiency. After the data is written to the LogBlock, the frontend device completes the write operation. Since append-only writes consume significant storage space, the backend device will write the data in the LogBlock to the VolBlock randomly after a certain period. This process of writing to the VolBlock is also called data replay. After data replay is complete, the data in the LogBlock is deleted to save storage space. LogBlock and VolBlock are data blocks used to store data; for example, they can be storage areas on the disk of the backend device. LogBlock uses a three-replica method to store data and can only perform append-only writes; VolBlock uses a three-replica method or EC (erasure coding) to store data and can perform random writes. Additionally, if data in a LogBlock is being written to a VolBlock, then the VolBlock is said to be in a playback state.

[0003] In related technologies, if VolBlock uses EC (Extended EC) to store data, the data written to LogBlock needs to be a full stripe to ensure the data playback process. For example, eight complete data objects need to be written. Therefore, when the front-end device writes data to the back-end device, it checks whether the data to be written is a full stripe. If so, the data to be written is transmitted to the back-end device so that the back-end device can write the data to LogBlock. If not, the front-end device will perform stripe alignment processing on the data to be written, that is, adjust the data to be written to a full stripe to obtain intermediate data, and then transmit the intermediate data to the back-end device so that the back-end device can write the intermediate data to LogBlock.

[0004] It is evident that if VolBlock uses EC mode to store data, the front-end device needs to perform stripe alignment processing on the data to be written when the data to be written indicated by the write operation command is not a full stripe. Stripe alignment processing is a relatively time-consuming process, which undoubtedly causes a large write operation delay. Summary of the Invention

[0005] The purpose of this invention is to provide a data writing method, apparatus, and electronic device to solve the latency problem of write operations while ensuring data playback. The specific technical solution is as follows:

[0006] In a first aspect, embodiments of the present invention provide a data writing method applied to a backend device; the method includes:

[0007] When the target storage block meets the data playback conditions, the target data to be played back in the target storage block is determined; wherein, the target storage block is a first type of storage block that writes data using the append write method;

[0008] Determine whether the target data is a full stripe; if not, perform stripe alignment processing on the target data in the target storage block to obtain aligned data.

[0009] The aligned data is written into a designated storage block corresponding to the target data storage location. The designated storage block is a second type of data block that uses a random write method to write data.

[0010] Optionally, the step of performing stripe alignment processing on the target data in the target storage block to obtain aligned data includes:

[0011] From each of the first and second type storage blocks of the backend device, locate the storage location of associated data related to the target data; wherein, the associated data includes the original data corresponding to the target data;

[0012] Read the associated data from the found storage location;

[0013] Using the associated data, the target data in the target storage block is adjusted to a full stripe to obtain aligned data.

[0014] Optionally, adjusting the target data in the target storage block to full stripe using the associated data to obtain aligned data includes:

[0015] Using the target data, replace the original data included in the associated data to obtain intermediate data;

[0016] The target data is deleted from the target storage block, and the intermediate data is written into the target storage block to obtain aligned data.

[0017] Optionally, reading the associated data from the found storage location includes:

[0018] If the found storage location belongs to a location in the first type of storage block, then the associated data is read from the storage location;

[0019] If the found storage location belongs to a location in the second type of storage block, then check whether the associated data in the storage location is in a playback state;

[0020] If so, monitor the status of the associated data, and when the status of the associated data changes to indicate that playback is complete, read the associated data from the storage location; otherwise, read the associated data from the storage location.

[0021] Optionally, the method further includes:

[0022] When a read operation instruction is received, locate the storage block where the data to be read, as indicated by the read operation instruction, is located;

[0023] If the storage block where the data to be read is located is a second type of storage block, determine the state of the data to be read;

[0024] If the data to be read is in playback mode, monitor the status of the data to be read.

[0025] If the state of the data to be read changes to a state indicating that playback is complete, then the data to be read is read.

[0026] Secondly, embodiments of the present invention provide a data writing device applied to a backend device; the device includes:

[0027] The determination module is used to determine the target data to be replayed in the target storage block when the target storage block meets the data replay conditions; wherein, the target storage block is a first type of storage block that writes data using an append write method;

[0028] The judgment module is used to determine whether the target data is a full stripe;

[0029] An alignment module is used to perform stripe alignment processing on the target data in the target storage block, if not, to obtain aligned data;

[0030] The writing module is used to write the aligned data into a designated storage block corresponding to the target data storage location. The designated storage block is a second type of data block that uses a random write method to write data.

[0031] Optionally, the alignment module includes:

[0032] The lookup submodule is used to find the storage location of associated data related to the target data from each of the first type of storage blocks and the second type of storage blocks of the backend device; wherein, the associated data includes the original data corresponding to the target data;

[0033] The read submodule is used to read the associated data from the found storage location;

[0034] The adjustment submodule is used to adjust the target data in the target storage block to a full stripe using the associated data, thereby obtaining aligned data.

[0035] Optionally, the adjustment submodule includes:

[0036] The replacement unit is used to replace the original data included in the associated data with the target data to obtain intermediate data;

[0037] The rewrite unit is used to delete the target data from the target storage block and write the intermediate data into the target storage block to obtain aligned data.

[0038] Optionally, the reading submodule includes:

[0039] The first reading subunit is used to read the associated data from the storage location if the found storage location belongs to the location in the first type of storage block;

[0040] The detection subunit is used to detect whether the associated data in the storage location is in a playback state if the found storage location belongs to a location in the second type of storage block.

[0041] The second reading subunit is configured to monitor the status of the associated data if the status changes to a state indicating completion of playback, and then read the associated data from the storage location; otherwise, read the associated data from the storage location.

[0042] Optionally, the device further includes:

[0043] The lookup module is used to locate the storage block where the data to be read, indicated by the read operation instruction, is located when a read operation instruction is received.

[0044] The status recognition module is used to determine the status of the data to be read if the storage block where the data to be read is located is a second type of storage block;

[0045] The monitoring module is used to monitor the status of the data to be read if the data to be read is in a playback state;

[0046] The reading module is used to read the data to be read when the state of the data to be read changes to a state indicating that playback is complete.

[0047] Thirdly, embodiments of the present invention provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0048] Memory, used to store computer programs;

[0049] When a processor executes a program stored in memory, it implements any of the data writing method steps described above.

[0050] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the data writing method steps described above.

[0051] Beneficial effects of the embodiments of the present invention:

[0052] The solution provided in this invention involves the backend device determining the target data to be replayed in the target storage block when the target storage block meets the data playback conditions. Then, it determines whether the target data is a full stripe. If not, it performs stripe alignment processing on the target data in the target storage block to obtain aligned data. Next, it writes the aligned data into a designated storage block corresponding to the target data storage location to complete the playback of the target data. Since the stripe alignment processing is performed during the playback process of the backend device, the frontend device does not need to perform stripe alignment processing for incomplete stripes and can directly write the data to the backend device, thereby reducing the latency of write operations. Therefore, this solution can solve the latency problem of write operations while ensuring data playback.

[0053] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0055] Figure 1 A flowchart of a data writing method provided in an embodiment of the present invention;

[0056] Figure 2 This is another flowchart of a data writing method provided in an embodiment of the present invention;

[0057] Figure 3This is another flowchart of a data writing method provided in an embodiment of the present invention;

[0058] Figure 4 A flowchart of a response to a write operation instruction provided in an embodiment of the present invention;

[0059] Figure 5 A schematic diagram illustrating a data merging method provided in an embodiment of the present invention;

[0060] Figure 6 A flowchart of a response to a read operation command provided in an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the structure of a data writing device provided in an embodiment of the present invention;

[0062] Figure 8 This is another structural schematic diagram of a data writing device provided in an embodiment of the present invention;

[0063] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0065] Because append-only writes are highly efficient, when a backend device receives a write command from the frontend device, it appends the data specified in the command to the LogBlock to improve write efficiency. Once the data is written to the LogBlock, the frontend device considers the write operation complete. Since append-only writes consume significant storage space, the backend device will rewrite the data from the LogBlock to a VolBlock after a certain period of time, and then delete the data from the LogBlock after the replay is complete to save storage space.

[0066] In related technologies, if VolBlock uses EC encoding to store data, the data written to LogBlock needs to be full-striped to ensure data playback. If the data to be written is not full-striped, the front-end device needs to perform stripe alignment on the data before writing it to LogBlock, thus completing the write operation. After the stripe-aligned data is written to LogBlock, when playing back the data in LogBlock, the back-end device first performs EC encoding on the data in LogBlock, and then writes the data to an EC-type VolBlock, thereby completing the data playback process.

[0067] Since stripe alignment is performed on the front-end device, when the front-end device writes data to the back-end device through the read / write interface, the front-end device will determine whether the data to be written is a full stripe. If it does not meet the requirement, stripe alignment will be performed, and the stripe-aligned data will be stored in LogBlock. In this way, when the data is replayed later, the stripe-aligned data can be EC encoded and then written to VolBlock.

[0068] It's important to note that a stripe is a sequence of data objects of the same size. For example, a stripe can consist of 6, 8, or 10 data objects. If a stripe consists of 8 data objects, then writing all 8 data objects constitutes a full stripe; otherwise, it's a partial stripe. When storing data in stripe form, data is written sequentially to each data object within the stripe. Once a stripe is full, i.e., when several complete data objects have been written, the next stripe is moved on. During storage, multiple data objects within a stripe are stored on different disks, with each disk storing a portion of the stripe. The size of the data objects within a stripe can be pre-configured, typically set to 2MB to 4MB.

[0069] When a front-end device needs to modify data stored in a back-end device—that is, when the front-end device sends a write operation command to the back-end device for data update—the data to be written is often not a full stripe. During stripe alignment, the front-end device reads the associated data from the LogBlock and VolBlock in the back-end device. This associated data includes the original data corresponding to the data to be written, and this associated data is a full stripe. Then, the original data in the associated data is replaced with the data to be written, forming intermediate data. This intermediate data is a full stripe, thus completing the stripe alignment. Finally, the stripe-aligned data is written to the LogBlock, completing the write operation.

[0070] However, during stripe alignment, if the read associated data is stored in an EC type VolBlock and is currently being replayed, it needs to wait for the replay to complete before being read again. Then, the data to be written replaces the original data in the associated data, forming intermediate data. Therefore, when the data written from the front-end device to the back-end device is not a full stripe, stripe alignment via the front-end device causes significant write operation latency.

[0071] To address the latency issue of write operations while ensuring data playback, embodiments of the present invention provide a data writing method, apparatus, and electronic device.

[0072] The following section first introduces a data writing method provided by an embodiment of the present invention.

[0073] The data writing method provided in this embodiment of the invention is applied to a backend device that communicates with a frontend device. For example, the backend device can be a server in a distributed storage system used to store data resources; the frontend device can be a client device. The frontend device can send read and write operation instructions to the backend device through a read / write interface. After receiving the read and write operation instructions from the frontend device, the backend device performs the corresponding read and write operations and returns the operation results to the frontend device after the operations are completed. The write operation instructions can include: write operation instructions for adding data, i.e., instructions for writing new data to the backend device; or write operation instructions for updating data, i.e., instructions for updating the data in the backend device.

[0074] It should be noted that, in the solution provided in the embodiments of the present invention, the data block written using the append-only method is referred to as the first type of storage block, while the data block written using the random write method is referred to as the second type of storage block. For example, the first type of storage block can be the LogBlock mentioned above, and the second type of storage block can be the VolBlock mentioned above, but it is not limited to these.

[0075] Furthermore, the second type of storage block is an EC type storage block or a three-replica format, meaning that data is stored in the second type of storage block in EC mode or in a three-replica format. The solution involved in this invention is provided for the case where the second type of storage block is an EC type data block. Moreover, the data stored in the first type of storage block can be three-replica data; however, provided that append writes are implemented, the embodiments of this invention do not limit the number of replicas.

[0076] The data writing method provided in this embodiment of the invention may include the following steps:

[0077] When the target storage block meets the data playback conditions, the target data to be played back in the target storage block is determined; wherein, the target storage block is a first type of storage block that writes data using the append write method;

[0078] Determine whether the target data is a full stripe;

[0079] If not, perform stripe alignment on the target data in the target storage block to obtain aligned data;

[0080] The aligned data is written into a designated storage block corresponding to the target data storage location. The designated storage block is a second type of data block that uses a random write method to write data.

[0081] In the solution provided by this invention, when the target storage block meets the data playback conditions, the backend device determines the target data to be played back in the target storage block, and then determines whether the target data is a full stripe. If not, stripe alignment processing is performed on the target data in the target storage block to obtain aligned data. Next, the aligned data is written to the designated storage block corresponding to the target data storage location to complete the playback of the target data. Since the stripe alignment processing is performed during the playback process of the backend device, the frontend device does not need to perform stripe alignment processing for incomplete stripes and can directly write the data to the backend device, thereby reducing the latency of write operations. Therefore, this solution can solve the latency problem of write operations while ensuring data playback.

[0082] The data writing method provided by the embodiments of the present invention will be described below with reference to the accompanying drawings.

[0083] like Figure 1 As shown, the data writing method provided in this embodiment of the invention is applied to a backend device; the method may include steps S101-S104:

[0084] S101, when the target storage block meets the data playback conditions, determine the target data to be played back in the target storage block; wherein, the target storage block is a first type of storage block that uses the append write method to write data;

[0085] In this embodiment, the target storage block is a first type of storage block that writes data using an append-only method. Since the append-only method is more efficient but consumes more storage space, when any first type of storage block meets the data playback conditions, the data in the first type of storage block can be written to the second type of storage block in a random write manner. After the playback is completed, the data in the first type of storage block is deleted to save storage space.

[0086] The aforementioned data playback conditions could be that any of the first-class storage blocks is full of data, or that the storage time of any of the first-class storage blocks reaches a time threshold preset by the staff, etc.

[0087] The method for determining the target data can be as follows: Select the data that has not been replayed from the target storage block as the target data. It is understood that each time data is written to the target storage block, the written data may be smaller than the storage space size of the target storage block. Thus, the target storage block can store data written at least once, i.e., data written in response to at least one write operation instruction. When the target storage block meets the replay conditions, there can be one or more target data items to be replayed, and each target data item can be data written in response to a write operation instruction.

[0088] S102, determine whether the target data is a full stripe;

[0089] Understandably, since the data replayed into the EC type type 2 storage block needs to be full-striped, after determining the target data through step S101, it can be determined whether the target data is full-striped. If it is not full-striped, the subsequent step S103 is executed. In this way, the full-strip analysis and subsequent stripe alignment processing are both performed by the back-end device. This means that for various write operation commands, the front-end device does not need to care whether the data to be written is full-striped, and when it is not full-striped, it does not need to perform stripe alignment processing for the incomplete stripe; it can directly write the data to the back-end device, thereby reducing the latency of write operations.

[0090] Since write operation instructions used for data updates modify the data stored in the backend device, the data to be written indicated by the write operation instructions for data updates is usually not a full stripe, resulting in a large write latency when the data to be written is stored in the first type of storage block according to the relevant technology; and the data to be written indicated by the write operation instructions for data addition is usually a full stripe.

[0091] Based on this, for example, in one implementation, determining whether the target data is a full stripe can specifically include: if the target data is data written in response to a specified instruction, determining whether the target data is a full stripe. In this implementation, for the specified instruction, since the backend device can determine whether the data is a full stripe during data playback and perform stripe alignment when the data is not full, the frontend device does not need to concern itself with whether the data to be written meets the stripe requirement, thus eliminating the need for stripe alignment.

[0092] For example, identifying whether the target data is data written in response to a specified instruction may include: based on the current storage location of the target data and the mapping relationship between various storage locations stored in the naming server, searching for a storage location in a second type of storage block that has a mapping relationship with the current storage location of the target data; if there is data in the storage location of the second type of storage block, then the target data is data written in response to a specified instruction.

[0093] It should be noted that in a distributed storage system, a naming server can be pre-established. This naming server is used to store the mapping relationship between the storage location of data in the client, the storage location in each first-class storage block, and the storage location in each second-class storage block, i.e., the mapping relationship between the various storage locations mentioned above.

[0094] S103, if not, perform stripe alignment on the target data in the target storage block to obtain aligned data;

[0095] Understandably, since the target data to be replayed into the second type of EC storage block needs to be full stripe, if the target data is not full stripe, it needs to be stripe aligned, that is, padded to be full stripe, in order to obtain aligned data.

[0096] To ensure clarity of the solution and layout, subsequent descriptions, in conjunction with other embodiments, will explain how to perform stripe alignment processing on the target data in the target storage block.

[0097] S104, the aligned data is written into the specified storage block corresponding to the target data storage location. The specified storage block is a second type of data block that uses a random write method to write data.

[0098] Understandably, for the front-end device, the data writing process involves writing data to various storage partitions within a volume on the front-end device. Since the name server stores the mapping relationships between the data storage locations on the front-end device, the storage locations within each first-type storage block, and the storage locations within each second-type storage block, after obtaining the alignment data in step S103, the storage location within the specified storage block can be found based on the target data's storage location and the mapping relationships stored in the name server, and the alignment data can be written to that specified storage block. This specified storage block is a second-type data block where data is written using a random write method, thus enabling data playback.

[0099] In the solution provided in this embodiment, when the target storage block meets the data playback conditions, the backend device determines the target data to be played back in the target storage block, and then determines whether the target data is a full stripe. If not, stripe alignment processing is performed on the target data in the target storage block to obtain aligned data. Next, the aligned data is written to the designated storage block corresponding to the target data storage location to complete the playback of the target data. Since the stripe alignment processing is performed during the playback process of the backend device, the frontend device does not need to perform stripe alignment processing for incomplete stripes and can directly write the data to the backend device, thereby reducing the latency of write operations. Therefore, this solution can solve the latency problem of write operations while ensuring data playback.

[0100] Alternatively, in another embodiment of the invention, in Figure 1 Based on the embodiments shown, such as Figure 2 As shown, stripe alignment is performed on the target data in the target storage block to obtain aligned data, including steps S1031-S1033:

[0101] S1031, Locate the storage location of associated data related to the target data from each of the first and second type storage blocks of the backend device; wherein, the associated data includes the original data corresponding to the target data;

[0102] For example, the method for finding the storage location of associated data related to the target data can be to sequentially search for the storage location of associated data from the first and second type storage blocks that have a mapping relationship with the current storage location of the target data, based on the mapping relationship between the storage location of the data in the client stored in the naming server, the storage locations in each first type storage block, and the storage locations in each second type storage block. The associated data is either the data in the storage location of the first type storage block corresponding to the storage location of the target data, or the data in the storage location of the second type storage block corresponding to the storage location of the target data.

[0103] It is understandable that the associated data is the data that needs to be updated. This associated data includes the original data corresponding to the target data, that is, the original data that needs to be replaced by the target data.

[0104] S1032, Read the associated data from the found storage location;

[0105] After finding the storage location of the associated data in step S1031, the associated data is read. If the associated data is found in the storage location of the second type of storage block, it is necessary to read all the data objects in the stripe where the associated data is located and concatenate the data in all the data objects. The concatenated data is the associated data.

[0106] Optionally, in one implementation, the associated data is read from the found storage location, including steps A1-A3:

[0107] A1. If the found storage location belongs to a location in the first type of storage block, then read the associated data from the storage location;

[0108] It is understandable that data is written sequentially into the first type of storage blocks. When data is played back, the data in each of the first type of storage blocks is played back in the order of writing. If the target data is being played back, and if related data is found in any of the first type of storage blocks, the related data is the data that has completed the stripe alignment operation. At this time, the related data is read.

[0109] A2, If the found storage location belongs to the second type of storage block, then check whether the associated data in the storage location is in a playback state;

[0110] If the associated data is not found in any of the first-class storage blocks, the associated data is searched in the second-class storage blocks. If found, it is checked whether the associated data is in a playback state. The playback state indicates that in the stripe containing the associated data, some data objects have completed data playback, while the data objects of other data objects have not completed data playback.

[0111] It's important to note that during data playback, an identifier is created to indicate the current playback status. If all data objects in the strip containing the related data have completed playback, the identifier indicates a completed playback state. If some data objects in the strip have completed playback while others have not, the identifier indicates a playback state. Therefore, when searching for related data, this identifier can be used to determine whether the related data is in a playback state.

[0112] A3, if yes, monitor the status of the associated data, and when the status of the associated data changes to indicate that playback is complete, read the associated data from the storage location; otherwise, read the associated data from the storage location.

[0113] For example, the backend device can use a thread pool to poll and monitor the status of the associated data. When the status of the associated data changes to a state indicating that playback is complete, that is, the data of each data object in the strip containing the associated data has been played back, and the associated data can be read.

[0114] If the associated data in the second type of storage block is not in a playback state, it can be read directly. The method for reading the associated data is as follows: read the data from all data objects in the stripe containing the associated data and concatenate them into complete associated data.

[0115] S1033, using the associated data, adjust the target data in the target storage block to full stripe to obtain aligned data.

[0116] After reading the associated data of the full stripe through step S1032, the target data is filled into a full stripe using the data in the associated data that does not need to be updated, and the aligned data can be obtained.

[0117] Optionally, in one implementation, the associated data is used to adjust the target data in the target storage block to full stripe to obtain aligned data, including steps B1-B2:

[0118] B1. Using the target data, replace the original data included in the associated data to obtain intermediate data;

[0119] It is understandable that by replacing the original data corresponding to the position of the target data in the associated data with the target data, the intermediate data of the full strip can be obtained.

[0120] B2, delete the target data from the target storage block and write the intermediate data into the target storage block to obtain the aligned data.

[0121] After obtaining intermediate data using target data and associated data, the target data that is not full of stripes in the target storage block is deleted. Then, the intermediate data that is full of stripes is written into the target storage block to obtain aligned data.

[0122] As can be seen, this solution utilizes associated data to adjust the target data to full stripe size. When the target data is subsequently played back, EC encoding can be directly performed, and then the alignment data is overwritten into the second type of storage block containing the associated data, thus completing the modification of the associated data. Furthermore, by using the associated data to stripe the target data and storing the resulting intermediate data in the target storage block, subsequent data retrieval can directly access the aligned data in the target storage block, which is the modified and correct data.

[0123] Alternatively, in another embodiment of the invention, in Figure 1 Based on the embodiments shown, such as Figure 3 As shown, the embodiments of the present invention further include steps S301-S304:

[0124] S301, When a read operation instruction is received, locate the storage block where the data to be read indicated by the read operation instruction is located;

[0125] When a backend device receives a read operation command from a frontend device, it can locate the storage block containing the information in the read operation command that indicates the storage location of the data to be read in the frontend device. For example, when the backend device receives the information from the frontend device indicating the storage location of the data to be read, it searches the mapping relationship stored in the naming server for the storage block that has a mapping relationship with the storage location of the data to be read in the frontend device. This storage block is the storage block where the data to be read, as indicated by the read operation command, is located.

[0126] S302, if the storage block where the data to be read is located is a second type of storage block, determine the status of the data to be read;

[0127] When reading data, the data to be read may not be a full stripe. If the data to be read is located in a second type of storage block, when reading, the entire stripe containing the data to be read needs to be read. That is, all the data in the stripe containing the data to be read can be read. The data to be read can exist in several data objects in the stripe.

[0128] In a distributed storage system, each data object in a stripe is stored in a Type II storage block on different disks. When data to be read from a Type II storage block needs to be read, all data in the stripe containing that data needs to be read. Because the data objects in a stripe are distributed across different Type II storage blocks on different disks, during the read operation, it can be determined whether the data to be read is in a playback state, i.e., whether the stripe containing the data to be read is in a playback state.

[0129] S303, If the data to be read is in playback mode, monitor the status of the data to be read;

[0130] It is understandable that, due to the different write speeds of various disks or the different network speeds during reading, when reading data from various data objects in the stripe where the data to be read is located, some data objects may have completed playback, while data from other data objects may not have completed playback. In this case, the state of the stripe where the data to be read is located is the playback state.

[0131] If the stripe containing the data to be read is in a playback state, the status of the data can be monitored. For example, when reading all data in the stripe containing the data to be read, if the stripe is in a playback state, the status of the data to be read is detected by polling the thread pool.

[0132] S304, if the state of the data to be read changes to a state indicating that playback is complete, then the data to be read is read.

[0133] When the state of the data to be read changes to a state indicating completion of playback, meaning that all data objects in the strip containing the data to be read have completed data playback, the data to be read is then read. For example, the method for reading the data to be read could be: when the thread pool detects that the state of the data to be read has changed to a state indicating completion of playback, it reads data from all data objects in the strip containing the data to be read, concatenates them into complete data, and then reads the data to be read from this complete data.

[0134] In related technologies, when a front-end device needs to read data, it sequentially reads data from each first-class storage block and each second-class storage block. If the data to be read is located in a second-class storage block and is in a playback state, a read error may occur. This is because reading data from a second-class storage block requires retrieving all data objects from a stripe, and these data objects are stored separately in each second-class storage block. If the data to be read is in a playback state, due to differences in data write speeds between the second-class storage blocks or differences in network speeds during reading, some data objects stored in the second-class storage blocks may have completed playback while others have not. Therefore, when concatenating the data objects from the second-class storage blocks to form the data to be read, incorrect data will be generated.

[0135] As can be seen, through this embodiment, if the data to be read is in a playback state, the state of the data to be read is monitored. When the state of the data to be read changes to a state indicating that the playback is complete, the operation of reading the data to be read is performed. This avoids the situation where the data to be read is in a playback state and reads the wrong data, thereby solving the problem of read-write inconsistency.

[0136] In addition, based on the above data writing method, from the perspective of the front-end device, this embodiment of the invention also provides a data writing method.

[0137] A data writing method provided from the perspective of the front-end device may include the following steps:

[0138] Once a write operation is detected, a write operation instruction is generated based on the write operation.

[0139] The write operation instruction is sent to the backend device so that the backend device responds to the write operation instruction and writes the data to be written carried by the write operation instruction into the first type of storage block.

[0140] Specifically, if the write operation is for updating data, a write operation instruction for updating data is generated; if the write operation is for adding data, a write operation instruction for adding data is generated. The method provided in this embodiment allows the front-end device to directly write data to the back-end device when writing data to the first type of storage block, without needing to consider whether the data to be written is full-striped, thus eliminating the need for stripe alignment processing when the stripe is not full. Therefore, it can reduce write operation latency while ensuring data playback.

[0141] To better understand the embodiments of the present invention, the following will be combined with... Figure 4 , Figure 5 and Figure 6 A specific example of the present invention will be described.

[0142] (1) Data writing stage: such as Figure 4 As shown, when a client device (corresponding to the front-end device mentioned above) writes data to the server (corresponding to the back-end device mentioned above) through the read / write interface, the data is directly written to the main LogBlock (corresponding to the first type of storage block mentioned above) in the server. After the data is written to the main LogBlock, the data in the main LogBlock is synchronized to the two slaves, which is considered to complete the write operation. For the client device, there is no need to care whether the written data is full stripe or to perform stripe alignment, thereby reducing the latency of the write operation.

[0143] (2) Data replay stage: When the server replays the target data in the LogBlock into the EC type VolBlock (corresponding to the second type of storage block mentioned above), it needs to determine whether the target data to be replayed is a full stripe. If not, it first needs to read out the associated data related to the target data in the EC type VolBlock. This associated data is a full stripe, and it includes the original data to be replaced by the target data.

[0144] Next, the retrieved correlated data is merged with the target data in the LogBlock. This involves replacing the portion of the original data corresponding to the target data in the correlated data with the target data. The merged data is then rewritten into the LogBlock, thus completing the stripe alignment. Figure 5As shown, a stripe contains 8 data objects. During data merging, the data of all data objects in the VolBlock is read first. Then, the target data in the LogBlock replaces the corresponding position in the VolBlock, and the resulting new data (corresponding to the intermediate data mentioned above) is stored in the LogBlock, thus completing the stripe alignment. At this point, the data in the LogBlock is the aligned data.

[0145] Next, the server synchronizes the data in the LogBlock, writing the aligned data from that LogBlock to the other two slaves. Then, the LogBlock is replayed again, overwriting the striped aligned data into the EC type VolBlock. This completes the replay process. Since the entire replay process occurs on the server, it is invisible to the client device, reducing the latency of write operations on the client device, making it suitable for write-heavy, read-light applications. Furthermore, it decouples read and write operations, making maintenance easier for operations personnel.

[0146] (3) Data reading phase: When the client device initiates a read operation, such as Figure 6 As shown, data in the main LogBlock will be read first, followed by data in other LogBlocks in a predetermined order. Reading data from an EC type VolBlock involves three cases:

[0147] The first scenario is that if the data being read is being replayed and the replay operation has already re-aligned the data and written it to the LogBlock, then the data read from the LogBlock can be returned directly without continuing to read the EC type VolBlock. This reduces the EC read process in this data reading, greatly improves the reading efficiency, reduces the reading latency, and avoids the situation of reading incorrect data when reading and replaying are performed simultaneously.

[0148] The second scenario is if the data being read has already been replayed into an EC type VolBlock. In this case, the data needs to be read from all LogBlocks in a predetermined order. When the reading reaches a VolBlock, the EC read process is performed, and then the data to be read is returned.

[0149] The third scenario is that the read data is not replayed into an EC type VolBlock. In this case, the data can be read directly from all LogBlock levels, and then the data that needs to be read can be returned.

[0150] As can be seen, this solution can effectively solve the problem of data reading errors when reading and playback are performed simultaneously, and can also solve the problem of excessive write operation latency caused by data alignment on the front-end device side.

[0151] Corresponding to the above method embodiments, this invention also provides a data writing device applied to a backend device; such as Figure 7 As shown, the device includes:

[0152] The determination module 710 is used to determine the target data to be replayed in the target storage block when the target storage block meets the data replay conditions; wherein, the target storage block is a first type of storage block that writes data using an append write method;

[0153] The judgment module 720 is used to determine whether the target data is a full stripe;

[0154] Alignment module 730 is used to perform stripe alignment processing on the target data in the target storage block if no, to obtain aligned data;

[0155] The writing module 740 is used to write the aligned data into a designated storage block corresponding to the target data storage location, wherein the designated storage block is a second type of data block that uses a random write method to write data.

[0156] Optionally, the alignment module includes:

[0157] The lookup submodule is used to find the storage location of associated data related to the target data from each of the first type of storage blocks and the second type of storage blocks of the backend device; wherein, the associated data includes the original data corresponding to the target data;

[0158] The read submodule is used to read the associated data from the found storage location;

[0159] The adjustment submodule is used to adjust the target data in the target storage block to a full stripe using the associated data, thereby obtaining aligned data.

[0160] Optionally, the adjustment submodule includes:

[0161] The replacement unit is used to replace the original data included in the associated data with the target data to obtain intermediate data;

[0162] The rewrite unit is used to delete the target data from the target storage block and write the intermediate data into the target storage block to obtain aligned data.

[0163] Optionally, the reading submodule includes:

[0164] The first reading subunit is used to read the associated data from the storage location if the found storage location belongs to the location in the first type of storage block;

[0165] The detection subunit is used to detect whether the associated data in the storage location is in a playback state if the found storage location belongs to a location in the second type of storage block.

[0166] The second reading subunit is configured to monitor the status of the associated data if the status changes to a state indicating completion of playback, and then read the associated data from the storage location; otherwise, read the associated data from the storage location.

[0167] Optionally, such as Figure 8 As shown, the device further includes:

[0168] The lookup module 810 is used to look up the storage block where the data to be read indicated by the read operation instruction is located when a read operation instruction is received.

[0169] The status recognition module 820 is used to determine the status of the data to be read if the storage block where the data to be read is located is a second type of storage block;

[0170] The monitoring module 830 is used to monitor the status of the data to be read if the data to be read is in a playback state;

[0171] The reading module 840 is used to read the data to be read when the state of the data to be read changes to a state indicating that playback is complete.

[0172] This invention also provides an electronic device, such as... Figure 9 As shown, it includes a processor 901, a communication interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.

[0173] Memory 903 is used to store computer programs;

[0174] When the processor 901 executes the program stored in the memory 903, it implements the steps of the data writing method described in any of the above embodiments.

[0175] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0176] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0177] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0178] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0179] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program that, when executed by a processor, implements the steps of any of the data writing methods described in the above embodiments.

[0180] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform the steps of any of the data writing methods described in the above embodiments.

[0181] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0182] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0183] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0184] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A data writing method, characterized in that, Applied to backend devices; the method includes: When the target storage block meets the data playback conditions, the target data to be played back in the target storage block is determined; wherein, the target storage block is a first type of storage block that writes data using the append write method; Determine whether the target data is a full stripe; If not, perform stripe alignment on the target data in the target storage block to obtain aligned data; The aligned data is written into a designated storage block corresponding to the target data storage location. The designated storage block is a second type of data block that uses a random write method to write data.

2. The method according to claim 1, characterized in that, The step of performing stripe alignment processing on the target data in the target storage block to obtain aligned data includes: From each of the first and second type storage blocks of the backend device, locate the storage location of associated data related to the target data; wherein, the associated data includes the original data corresponding to the target data; Read the associated data from the found storage location; Using the associated data, the target data in the target storage block is adjusted to a full stripe to obtain aligned data.

3. The method according to claim 2, characterized in that, The step of adjusting the target data in the target storage block to full stripe using the associated data to obtain aligned data includes: Using the target data, replace the original data included in the associated data to obtain intermediate data; The target data is deleted from the target storage block, and the intermediate data is written into the target storage block to obtain aligned data.

4. The method according to claim 2, characterized in that, Reading the associated data from the found storage location includes: If the found storage location belongs to a location in the first type of storage block, then the associated data is read from the storage location; If the found storage location belongs to a location in the second type of storage block, then check whether the associated data in the storage location is in a playback state; If so, monitor the status of the associated data, and when the status of the associated data changes to indicate that playback is complete, read the associated data from the storage location; otherwise, read the associated data from the storage location.

5. The method according to claim 1, characterized in that, The method further includes: When a read operation instruction is received, locate the storage block where the data to be read, as indicated by the read operation instruction, is located; If the storage block where the data to be read is located is a second type of storage block, determine the state of the data to be read; If the data to be read is in playback mode, monitor the status of the data to be read. If the state of the data to be read changes to a state indicating that playback is complete, then the data to be read is read.

6. A data writing device, characterized in that, Applied to backend equipment; the device includes: The determination module is used to determine the target data to be replayed in the target storage block when the target storage block meets the data replay conditions; wherein, the target storage block is a first type of storage block that writes data using an append write method; The judgment module is used to determine whether the target data is a full stripe; An alignment module is used to perform stripe alignment processing on the target data in the target storage block, if not, to obtain aligned data; The writing module is used to write the aligned data into a designated storage block corresponding to the target data storage location. The designated storage block is a second type of data block that uses a random write method to write data.

7. The apparatus according to claim 6, characterized in that, The alignment module includes: The lookup submodule is used to find the storage location of associated data related to the target data from each of the first type of storage blocks and the second type of storage blocks of the backend device; wherein, the associated data includes the original data corresponding to the target data; The read submodule is used to read the associated data from the found storage location; The adjustment submodule is used to adjust the target data in the target storage block to a full stripe using the associated data, thereby obtaining aligned data.

8. The apparatus according to claim 6, characterized in that, The device further includes: The lookup module is used to locate the storage block where the data to be read, indicated by the read operation instruction, is located when a read operation instruction is received. The status recognition module is used to determine the status of the data to be read if the storage block where the data to be read is located is a second type of storage block; The monitoring module is used to monitor the status of the data to be read if the data to be read is in a playback state; The reading module is used to read the data to be read when the state of the data to be read changes to a state indicating that playback is complete.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the data writing method steps of any one of claims 1-5.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the data writing method steps of any one of claims 1-5.

Images