Distributed block storage-based replica data storage method and device

Abstract

The invention relates to a distributed block storage-based copy data disk dropping method and device, and the method comprises the steps: writing a copy data block corresponding to a write request into a physical disk under the condition that the write request of the copy data block of a target copy is received, and the write request is used for requesting to drop the copy data block into the disk; a reply message is returned, and the reply message carries the copy length of the target copy written into the physical disk; under the condition that a read request for reading the target copy is received and the copy data block requested by the read request is located in the copy length of the target copy, returning the copy data block requested by the read request; and returning a prompt message according to a search result of the copy data block requested by the read request under the condition that the copy data block requested by the read request is located outside the copy length of the target copy. The technical problem of low data disk dropping efficiency is solved.

Description

Technical Field

[0001] The present application relates to the field of cloud services, and in particular to a method and device for storing replica data on a disk based on distributed block storage. Background Art

[0002] With the development of the Internet, new requirements have been put forward for the rapid storage of data in distributed block storage. In the existing technology, for the storage of replicas, the data is written into the original data area through the append-write mode, and the index is stored and compressed in a hierarchical, ordered, disk-oriented data structure tree (Log Structured Merge Tree, LSM Tree). Finally, the consistency protocol Raft protocol is connected, and the replica passively accepts the data append write initiated by the master replica.

[0003] However, there is a double writing problem in the prior art, that is, when data is written to the disk, the log is written first and then the data is written, resulting in low efficiency of data writing to the disk. Summary of the invention

[0004] The present application provides a method and device for storing replica data on a disk based on distributed block storage to solve the problem of low efficiency of storing data on a disk.

[0005] In the first aspect, the present application provides a method for storing replica data on a disk based on distributed block storage, comprising: upon receiving a write request for a replica data block of a target replica, writing the replica data block corresponding to the write request to a physical disk, wherein the write request is used to request that the replica data block be stored on the disk; returning a reply message, wherein the reply message carries the replica length of the target replica that has been written to the physical disk; upon receiving a read request to read the target replica, if the replica data block requested by the read request is within the replica length of the target replica, returning the replica data block requested by the read request; if the replica data block requested by the read request is outside the replica length of the target replica, returning a prompt message based on the search result of the replica data block requested by the read request.

[0006] In the second aspect, the present application provides a device for storing replica data on a disk based on distributed block storage, comprising: a write module, which is used to write the replica data block corresponding to the write request to the physical disk when a write request for the replica data block of the target replica is received, wherein the write request is used to request that the replica data block be stored on the disk; a reply module, which is used to return a reply message, wherein the reply message carries the replica length of the target replica that has been written to the physical disk; a read module, which is used to return the replica data block requested by the read request when a read request for reading the target replica is received and when the replica data block requested by the read request is within the replica length of the target replica; and when the replica data block requested by the read request is outside the replica length of the target replica, return a prompt message based on the search result of the replica data block requested by the read request.

[0007] As an optional example, the above-mentioned writing module includes: a writing unit, used to write the above-mentioned replica data blocks to the above-mentioned physical disk in a random writing order when receiving the above-mentioned write request; a determination unit, used to determine the length of multiple continuous and uninterrupted replica data blocks as the above-mentioned replica length starting from the first replica data block of the above-mentioned target replica.

[0008] As an optional example, the above-mentioned determination unit includes: a determination sub-unit, which is used to store the above-mentioned write request corresponding to the above-mentioned replica data block into the message confirmation queue when any replica data block is written to the above-mentioned physical disk, wherein the write request in the above-mentioned message confirmation queue is a request that has been received and written; when there are 1st to Nth write requests in the above-mentioned message confirmation queue and the N+1th write request is not included, the above-mentioned N is determined as the above-mentioned replica length.

[0009] As an optional example, the above-mentioned reading module includes: a reply unit, used to search for the replica data block requested by the above-mentioned read request on the target physical disk; if the replica data block requested by the above-mentioned read request is found, the above-mentioned replica data block is returned; if the replica data block requested by the above-mentioned read request is not found, searching for the replica data block requested by the above-mentioned read request from other physical disks, wherein the above-mentioned other physical disks are other physical disks on which the above-mentioned target replica is written; if the replica data block requested by the above-mentioned read request is found in the above-mentioned other physical disks, the above-mentioned replica data block is returned; if the replica data block requested by the above-mentioned read request is not found in the above-mentioned other physical disks, an error message is returned.

[0010] As an optional example, the above-mentioned device also includes: a copy supplement module, which is used to, after the above-mentioned target copy is written to the above-mentioned physical disk, when the copy supplement operation is triggered, determine the longest target copy among all target copies of other physical disks as the data source; set the above-mentioned data source to a write-prohibited state; and use the above-mentioned data source to perform the copy supplement operation on the above-mentioned target copy on the target physical disk.

[0011] As an optional example, the above-mentioned copy-filling module includes: a copy-filling unit, used to read data starting from the offset position of the above-mentioned data source, wherein the above-mentioned offset position is the position in the above-mentioned data source that is offset by the copy length of the above-mentioned target copy; writing the read data to the above-mentioned target copy until the length of the above-mentioned target copy is the same as that of the above-mentioned data source; and updating the copy length of the above-mentioned target copy.

[0012] As an optional example, the above-mentioned device also includes: an execution module, which is used to assign a first operation type code to the write request when a write request for a replica data block of a target replica is received; inserting the first operation type code into a lock-free queue, wherein the lock-free queue also includes a second operation type code, and the second operation type code is a code for internal operations of the data server used for disk storage; and executing the write request and the internal operation in the order of the operation type codes in the lock-free queue.

[0013] In the third aspect, the present application provides an electronic device, comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; at least one memory connected to the at least one bus, wherein the processor is configured to: upon receiving a write request for a replica data block of a target replica, write the replica data block corresponding to the write request to a physical disk, wherein the write request is used to request that the replica data block be written to the disk; return a reply message, wherein the reply message carries the replica length of the target replica that has been written to the physical disk; upon receiving a read request to read the target replica, if the replica data block requested by the read request is within the replica length of the target replica, return the replica data block requested by the read request; if the replica data block requested by the read request is outside the replica length of the target replica, return a prompt message based on the search result of the replica data block requested by the read request.

[0014] In a fourth aspect, the present application further provides a computer storage medium storing computer executable instructions, wherein the computer executable instructions are used to execute any of the above-mentioned methods for storing replica data based on distributed block storage in the present application.

[0015] Compared with the prior art, the above technical solution provided by the embodiment of the present application has the following advantages: the method provided by the embodiment of the present application, when data is written to disk, for each replica data block, when a write request is received, the data block is written to the physical disk. When replying to a message, the replica length of the replica is replied, so that when reading data, only the data within the replica length is read, and the data exceeding the replica length is not guaranteed to be successfully returned, so that it is no longer necessary to perform a double write operation when data is written to disk, that is, the operation of writing logs first and then writing data, thereby improving the efficiency of data writing to disk. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0019] Figure 1 A flowchart of a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0020] Figure 2 A system diagram of a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0021] Figure 3 An IO path diagram of a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0022] Figure 4 An interactive diagram of modules of a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0023] Figure 5 A periodic task graph for executing a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0024] Figure 6 A schematic diagram of a data-client write process of a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0025] Figure 7 A schematic diagram of a data-server write process of a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0026] Figure 8 A schematic diagram of a data-client read process of a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0027] Fig. 9 A schematic diagram of a data-server read process of a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0028] Fig.10 A flowchart of a dataserver processing adding a replica AddReplica in a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0029] Fig.11 An inheritance diagram of a method for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0030] Fig.12 A structural diagram of a device for storing replica data on a disk based on distributed block storage provided in an embodiment of the present application;

[0031] Fig.13 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0033] The disclosure below provides many different embodiments or examples to implement different structures of the present invention. In order to simplify the disclosure of the present invention, the parts and settings of specific examples are described below. Of course, they are only examples, and the purpose is not to limit the present invention. In addition, the present invention can repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed.

[0034] In order to solve the technical problem of low efficiency of data storage in the prior art, the present application provides a method for storing replica data on disk based on distributed block storage, which can achieve the effect of improving the efficiency of data storage on disk.

[0035] Figure 1 A flowchart of a method for storing replica data on a disk based on distributed block storage is provided in an embodiment of the present application. It includes:

[0036] S102, when receiving a write request for a replica data block of a target replica, writing the replica data block corresponding to the write request to a physical disk, wherein the write request is used to request that the replica data block be written to the disk;

[0037] S104, returning a reply message, wherein the reply message carries the copy length of the target copy that has been written to the physical disk;

[0038] S106-1, when a read request for reading a target replica is received, and when the replica data block requested by the read request is within the replica length of the target replica, returning the replica data block requested by the read request;

[0039] S106-2: When the replica data block requested by the read request is outside the replica length of the target replica, a prompt message is returned according to the search result of the replica data block requested by the read request.

[0040] The above-mentioned method for storing replica data on disk can be applied to the process of storing replica data on disk in elastically scalable distributed block storage (Elastic blockstore, EBS).

[0041] In distributed storage, in order to ensure that the business data on the physical server will not be affected when the physical server crashes, the business data is usually written into multiple copies, each of which is called a Replica copy. Different copies can be written to different servers. This ensures that the business data will not be lost and can be accessed normally when the server crashes. The above-mentioned target copy can be a copy of the business data, and the write to disk means writing the copy data from the server to the physical disk.

[0042] In order to implement the above-mentioned method of storing replica data on disk, a system can be built. The schematic diagram of the system is as follows: Figure 2 As shown in the figure, users perform read and write operations on replicas through the cloud platform and virtual machines at the user layer, generating input / output (IO) requests (read / write requests), the block controller at the distributed block storage logic layer schedules the read / write requests, the block server forwards the read / write requests, the data controller at the storage layer schedules the replica data, and the data server reads and writes the replica data.

[0043] In the above system, the server used for data placement is the data server, which is a server on the data path of the physical disk for single-node data IO request placement and resource management; the block server block-server is a server that calls the data-client interface to initiate IO requests to the data server where all replicas are located. It is responsible for accepting customer IO requests, applying for resources, issuing IO requests and establishing indexes; the block controller block-master is a server that controls the block-server; the data controller data-master is a server on the physical disk path that is responsible for all data server resource management, replica recovery and balancing; the data-client is a client that initiates writing to multiple replicas.

[0044] In the above system, the path diagram of the IO request initiated by the data-client is as follows Figure 3 As shown, an IO request is generated from a virtual machine, sent to a block server through a distributed block storage engine, and then forwarded to a data server. The data server then performs a disk write operation (write operation) or a read operation on the replica data according to the IO request.

[0045] The role of the above data server is not only to read and write replicas for IO requests, but also to carry out other functions. Figure 4 As shown, Figure 4 This is the interaction diagram between the data server and other modules. From the perspective of data flow and control flow, their functions are as follows:

[0046] In terms of data flow: the data server takes over the business IO of the block-server and implements the append write semantics; it receives requests from the data-master to complete the data of the replica after it is written to the disk and copies the data to the completed replica; it loads the data on the disk and replies to the location where the data is written to ensure that the data that has been replied to the upper layer will not be lost.

[0047] In terms of control flow, it has the following management functions: node management, that is, registering nodes with the data-master, responding to and completing node offline / query requests. Disk management, that is, responding to and implementing disk online / offline requests; counting disk iops (Input / Output Operations Per Second, that is, the number of read and write (I / O) operations per second) / throughput / latency / capacity; block management, that is, applying for blocks / releasing blocks on demand; responding to block query and other requests; recovery task management, that is, responding to data-master requests to make up copies / synchronize data; realizing data synchronization of copies after being dropped to disk or completed copies, to ensure that data can continue to be restored after restart.

[0048] For the upstream and downstream modules of the data server, from the data flow: the main upper-layer modules of the data server include the ebs server, which distributes the IO requests passed from the front end to multiple data servers (the database client software development kit (Database System client Software Development Kit, dbsclient SDK) can be called to implement this function); and the lower layer implements writing and reading data by calling the two interfaces UserSpaceBlock::Write and UserSpaceBlock::Read. From the control flow point of view: the data server reports disk and metadata related information to the data master upward; and calls the BlockSystemController::GetBlockSystemInfo interface downward to obtain disk information. In addition, the data server also needs to respond to the replicas, disk addition and deletion operations and node start and stop operations sent by the data master.

[0049] The data server implements the control flow function through the following sub-modules.

[0050] Disk manager module: responsible for monitoring and recording the following information of the disks in the system: storemanager{[storeid,status,capacity,chunks,units,userblocksystemFs*],…} When a physical disk is added or a node is started, if there are data blocks on the physical disk, all existing chunks are automatically loaded when the process starts, and their status and size are restored; the physical disk information is registered with the physical disk monitoring program, and the SPDK poller on the corresponding physical disk is started. When the physical disk is offline: the related replica service is shut down, and the SPDK poller thread on it is also closed.

[0051] Block manager module: responsible for managing the conversion of logical addresses to physical addresses at the single-machine, single-disk level. According to the storage engine design, it records single chunk information, mainly including: chunkmanager{[chunkid,status,size,storeid,userblocksystemFs*],…}; dispatcher submit&&in-order ack logical; ExpiredIOFlushPoller. In addition, this layer also needs to implement seq chunk sequential append write and in-order ack semantics; in order to ensure that IO timeouts that cannot wait for ack can be returned, it is also necessary to periodically check the timeout IO and feedback to the data-client.

[0052] Recover service: mainly used to generate, schedule and execute Block copy recovery tasks. The records of Block copy recovery tasks are as follows:

[0053] recover manager{[chunkid,task-status,src_id,src_size],…};

[0054] Among them, the task content of each replica recovery (data alignment) of the retry thread retrytaskpoller is as follows: [chunkId, status, size, store_id, UserBlockFs*,] Block replica recovery task scheduling failed recovery tasks will enter the retry task queue in FIFO order. Recover serice has a periodic task, which will periodically select tasks for data recovery. The main function of the Recover service is to implement alarm and repair services after the replica, disk, and node are damaged. On the one hand, when a hardware abnormality is detected, the data server quickly reports the abnormality; on the other hand, after selecting the appropriate node and disk, it responds to the Recover request of the database (Data Base System, DBS) master.

[0055] Manager service module: physical disk Disk / node Node control related services, mainly responding to DBSmaster management, control, query requests, and implementing specific functions. Recorded as follows: manage service{registerDS / AllockChunk / DeleteChunk / AddReplica / AlignData / Dis ableDisk / GetBlocksEnableDisk / Statistic}.

[0056] Heartbeat timer module: Periodically reports the disk, replica and other information of the current storage node to DBSmaster. The reporting time is when data-master comes to periodically query. The main reported data includes: 1. All disk information recorded by Diskmanager; 2. All chunk information recorded by Chunk manager.

[0057] Scheduler module: responsible for periodically starting tasks such as Update Server statistic / flush expired io / Handle recover task. Its main periodic tasks are as follows: Figure 5As shown. Among them: UcxServerNetMasterPolling and UcxServerMsgHandlerPolling are the access threads of the compensation server ucxserver, responsible for receiving requests and replying information; SpdkThreadPoolExecutor is the core working thread of the data server, responsible for executing the context connection of the data flow and control flow, and SpdkThreadPoolExecutor is responsible for driving the above three periodic tasks. When executing the copy data recovery task, the request is sent to the UcxClientNetPoller client network poller to pull data, which can avoid the impact on business IO. IO cheker check is implemented in ucxserver to avoid the situation where the copy in recovery provides reads and the old version of data provides reads, which may lead to reading illegal data. IO statistic is used to count indicators such as latency / iops / throughput in the IO process.

[0058] For the above data server, in terms of reading and writing, the copy data can be written to the physical disk through the read and write functions. Figure 6 The following is a schematic diagram of the write process of the data-client of the DBS database. The user initiates a request, applies for memory from the memory pool through the spdk thread, then queries the replica location, sends the IO request to the data master, and the data master sends the IO request to the corresponding data server. Figure 6 In the example, the IO request is sent to data server 1, data server 2, and data server 3 respectively, and responses are obtained. Finally, the response is fed back to the user and the memory is released. It should be noted that the order of sending requests 0-2 and the order of obtaining responses 0-2 in the figure are examples and do not constitute a limitation. After the data-client IO request is sent to the data server, the data server can write the target copy requested by the IO request to the physical disk. Figure 7The following is a write flow diagram of the data server of the DBS database. The write request is sent to the data server through the UcxServerMsgHandlerPolling thread, and the data server performs the write operation. Then the spkd thread checks the data block chunk to check whether it exists, whether the status is readable and writable, and whether the data address in the data block is continuous, whether the data exceeds the boundary, and whether it is an overwrite write. Then the request is sent to the user's distributed block system and the response is received, and the chunk is checked again to check whether it exists, whether the status is readable and writable, and whether the IO is continuous. When the data server writes data, it performs out-of-order writing, and after writing, it performs an orderly reply to the user. For the target replica, it can be composed of replica data blocks. When the data server writes the target replica to the physical disk, it writes each replica data block of the target replica to the physical disk. Since there may be a write failure of the replica data block, the dataserver must reply to the replica length of the written target replica after writing the target replica to the physical disk. Replica length: The replica side of the dataserver has determined the farthest position where data is continuously written to the disk without any holes in front. It refers to the length of continuous data written to the physical disk and takes effect. No holes means that the data is continuous and there is no data missing. For example, if the replica data blocks are 1-10, and the data blocks written to the physical disk and take effect are 1-6, 8, and 9, then the replica length is 6. If the replica data blocks written to the physical disk are 1-3, 4-8, and 10, then the replica length of the target replica written to the physical disk is 3. The consistency point of the replica length refers to the replica length reached by more than half of the replicas. It is defined as the length of the current replica returned to the data client after the IO of each replica is completed. The length that more than half of the replicas exceed or reach is called the consistency point. For example, if there are 3 replicas, and the replica length of two of them reaches 3, the consistency point is 3. After the data server completes writing the target replica, it replies to the block-server with the replica length and consistency point of the target replica. When the client requests to read data, it can initiate an IO request to read data, such as Figure 8 The following is a diagram of the read process of the data-client of the DBS database. The user initiates a request, applies for memory from the memory pool through the spdk thread, and then queries the replica location from the data master. After finding out which data server the replica is located on, the block-server sends the IO request to the data server. The block-server determines which data servers the client requests to read from through the data-master and determines a data server to send the request to, which then provides the data. Fig. 9The following is a read flow diagram of the data-server of the DBS database. The read request is sent to the data server through the UcxServerMsgHandlerPolling thread, and the data server performs the read operation. The spkd thread checks the data block cucnk to see if it exists and whether it is readable and writable. If it exists and is readable and writable, check whether the data address in the data block is continuous and whether the data exceeds the limit. The data server replies the read data to the user. Since the data server returns the replica length after writing the target replica, the data within the replica length requested by the client can be returned to the client, but the block-server does not guarantee that it can return the data outside the replica length requested by the client.

[0059] That is to say, the data server implements sequential append writing; in the reply after writing, it brings back the replica length to the data-client; the data-client replies to the upper layer that the write is successful; in order to improve the concurrency of writing, the data server implements out-of-order writing; in order to avoid the out-of-order reply that may cause the process of supplementing the replica or aligning the replica data to be extremely complicated, the data server replies in order according to the replica length (that is, only replies to the position of the current replica length). The data server where each replica is located records its own length and only accepts read requests that do not exceed this length range; during reading, if the data requested by the read request exceeds the replica data in the current data server, it will switch to another replica and try again; if the target replica is located on physical disks A and B, the replica length of the target replica on physical disk A is 10, and the replica length of the target replica on physical disk B is 8. When the data client requests replica data block 9 from physical disk B, physical disk B does not have this data, so the target replica on physical disk A is used to provide the data. For data that is not fed back to the upper layer during writing, the read request does not promise to be successfully returned. For the request queue with orderly replies for out-of-order writing. In order to achieve orderly reply for out-of-order writes, the data server introduces an ack queue internally. The IOs that are submitted to the physical disk later but completed earlier are queued in this ack queue. The data server will reply to the data-client only after all the previous IOs are written to the disk and there are no gaps.

[0060] In this embodiment, when data is written to disk, for each replica data block, when a write request is received, the data block is directly written to the physical disk, without writing the log first and then writing the data. When replying to a message, the replica length of the replica is replied, so when reading data, only the data with the replica length is read, and the data exceeding the replica length is not guaranteed to be successfully returned, so that there is no need to perform double write operations when data is written to disk, which improves the efficiency of data writing to disk.

[0061] As an optional example, when a write request for a replica data block of a target replica is received, writing the replica data block corresponding to the write request to the physical disk includes: when a write request is received, writing the replica data block to the physical disk in a random write order; starting from the first replica data block of the target replica, determining the length of multiple consecutive and uninterrupted replica data blocks as the replica length.

[0062] In this embodiment, when writing replica data blocks, they can be written out of order. That is, it is not necessary to write in the order in which the replica data blocks are received. Instead, after the replica data blocks are received in order, the replica data blocks are written to the physical disk in a random order. The random writing order is not limited. For example, if the receiving order of the replica data blocks is 123456, then the writing order is random, which can be 135426 or 265314. There may be a write failure, such as 263_51, where 4 is not written successfully. Out of order writing does not require waiting in the receiving order, and also avoids the problem of affecting the writing of subsequent replica data blocks when data writing fails. The replica length is the length of the continuous and uninterrupted data blocks in the written replica data blocks. For example, in 135426, all 6 replica data blocks are written successfully, and the replica length is 6, while in 263_51, 1-3 are written successfully, and 4 is not written successfully, so the replica length is 3.

[0063] As an optional example, starting from the first replica data block of the target replica, determining the length of multiple continuous and uninterrupted replica data blocks as the replica length includes: when any replica data block is written to the physical disk, storing the write request corresponding to the replica data block in a message confirmation queue, wherein the write requests in the message confirmation queue are requests that have been received and written; when there are 1st to Nth write requests in the message confirmation queue and do not include the N+1th write request, N is determined as the replica length.

[0064] In this embodiment, each replica data block of multiple replica data blocks of a target replica can be requested to be written to the physical disk by a write request. Each replica data block corresponds to a write request. The write requests can be sorted according to the order of the replica data blocks, with a precedence order. After receiving a write request, the replica data block corresponding to the write request can be written to the physical disk, and the corresponding write request can be stored in the message confirmation queue. The data output request in the message confirmation queue is the write request of the replica data block that was successfully written, and the write request of the data block that was not successfully written will not be stored in the message confirmation queue.

[0065] Since write requests have a sequence, the sequence can be expressed by numbers. When the message confirmation queue contains the first N write requests with complete numbers, it is confirmed that the first N data blocks have been written successfully and the copy length is N.

[0066] That is to say, in this embodiment, the reply can be made in a way of out-of-order writing and orderly reply. After the replica data block is written, the reply is not made immediately, but the write request is written into the message confirmation queue for orderly reply.

[0067] As an optional example, when the replica data block requested by the read request is outside the replica length of the target replica, returning a prompt message based on the search result of the replica data block requested by the read request includes: searching for the replica data block requested by the read request on the target physical disk; when the replica data block requested by the read request is found, returning the replica data block; when the replica data block requested by the read request is not found, searching for the replica data block requested by the read request from other physical disks, wherein the other physical disks are other physical disks on which the target replica is written; when the replica data block requested by the read request is found in other physical disks, returning the replica data block; when the replica data block requested by the read request is not found in other physical disks, returning an error message.

[0068] After the replica data block is written and the reply message is returned, the client can request access to the replica data block written to the physical disk. The client can initiate a read request, which can request access to any one or more data blocks of the target replica. When the replica data block requested by the client is stored on the physical disk, the data server can return the replica data block. If the replica data block requested by the read request is not on the physical disk, it is necessary to search for the replica data block of the target replica on other physical disks, and the corresponding data server will return it.

[0069] If the target replica is located on physical disk 1 and physical disk 2, the replica length of the target replica on physical disk 1 is 10, and the replica length of the target replica on physical disk 2 is 12. When the client requests replica data block 11 of the target replica, the client can send a read request to physical disk 1. Physical disk 1 does not have replica data block 11, so replica data block 11 must be obtained from physical disk 2 and returned to the client. The operation of finding replica data block 11 can be completed by the data-master.

[0070] As an optional example, after the target copy is written to the physical disk, the above method also includes: when the copy replenishment operation is triggered, determining the longest target copy among all target copies of other physical disks as the data source; setting the data source to a write-prohibited state; and using the data source to perform the copy replenishment operation on the target copy on the target physical disk.

[0071] In this embodiment, since the target copy may lose data when writing to the physical disk and cannot be written completely, and part of the copy may be lost due to the failure of the physical disk, the process of copy filling may be involved. Copy filling mainly supplements the data of other target copies based on the longest copy among the target copies. The target copies are located on different physical disks and may have different lengths. Take the longest copy as the data source copy, and fill in the other target copies to the length of the longest target copy. When filling the copy, the source copy is prohibited from writing. After the other copies are filled, the write-forbidden state can be lifted.

[0072] There are many scenarios that trigger the replication operation. For example, scenarios such as DropReplica discarding replicas, DisableStore disabling storage, KickServer responding to servers, and Balance balancing all require replication operations. It can be understood that when the number of replicas of a data block is insufficient, a new replica replication operation will be triggered. The replication operation pulls data from normal replicas. Whether the replica balance seq chunk or random chunk can disable writing to src; the length of the replica to be deleted after the data is completed cannot be longer than the replica used as the data source. During the replication process, the longer one is selected from multiple available replicas as the src data source. The write-disabled state needs to be persisted to prevent the request from performing IO on the replica with partially restored data when the state is lost after the data server is restarted; otherwise, data errors or data loss may occur.

[0073] When adding a replica, the data-master selects the data-sever to be added, and adds a replica to the data-master metadata (the replica status is kReplicaCreating); the data-master sends an AddReplica request to the selected data-sever; the data server processes AddReplica as follows: Fig.10As shown. data-master processes the return code of data-sever AddReplica; data-sever returns kNotFinish; data-master sleeps for 1 second and tries AddReplica again; data-sever returns kOk; data-master considers it successful and updates its metadata (updates the replica status to kReplicaActive); data-sever returns other error codes; data-master considers it a failure and updates its metadata (updates the replica status to kReplicaDeleting); then data-master sends a Deletechunk request to data-sever; then, when data-master checks and finds that the chunk replica data is insufficient, it selects data-sever again and performs AddReplica to make up the replica.

[0074] Specifically for the balancing and replica replenishment process of random / sequential blocks, the process of data-master initiating operations is as follows:

[0075] For random block balancing (src->data-server):

[0076] Create tasks a and b. Task a: AddReplica on data-server; Task b: Remove the replica on src source; Execute task a: Select src of AddReplica, data-server pulls data from src (data-sever will set src to read only state when pulling data for the first time). After pulling data, set the data-server replica state to active, and finally remove the src replica, and set the status of all remaining replicas to normal state.

[0077] For random block copy:

[0078] First, select data-server, select src of AddReplica, and then data-server pulls data from src (data-sever will set src to read only status when pulling data for the first time). After pulling the data, set the data-server replica status to active, and set the status of all remaining replicas to normal status.

[0079] For seq block copy:

[0080] First, select data-server, select src of AddReplica, and then data-server pulls data from src. After the data is pulled, set the data-server replica status to active.

[0081] For seq block balancing:

[0082] Create tasks a and b. Task a: AddReplica on data-server; Task b: Remove replica on src; Execute task a: Select src for AddReplica and set src to read_only state; data-server pulls data from src; After pulling data, set the data-server replica state to active, remove the src replica, and set the status of all remaining replicas to normal state.

[0083] As an optional example, using a data source to perform a copy-fill operation on a target copy on a target physical disk includes: reading data starting from an offset position of the data source, where the offset position is a position in the data source that is offset by the copy length of the target copy; writing the read data to the target copy until the length of the target copy and the data source are the same; and updating the copy length of the target copy.

[0084] In this embodiment, since the length of the data source is not consistent with the length of the target copy to be completed, and the first part of the data of the target copy to be completed is complete, when completing the target copy, data can be read from the migration position of the data source. If the copy length of the target copy to be completed is 10, it means that the first 10 copy data blocks of the target copy to be completed are complete, therefore, data is read from the 11th data block of the data source and completed to the target copy to be completed.

[0085] As an optional example, in the case of receiving a write request for a replica data block of a target replica, the above method also includes: assigning a first operation type code to the write request; inserting the first operation type code into a lock-free queue, wherein the lock-free queue also includes a second operation type code, and the second operation type code is a code for internal operations of the data server used for disk storage; and executing the write request and internal operations in the order of the operation type codes in the lock-free queue.

[0086] In this embodiment, for input and output requests (including read requests and write requests), they can be uniformly assigned operation type codes according to their types together with the operations inside the data server. The input and output requests and the operations inside the data server have unique operation type codes corresponding to them. The operation type codes are stored in the lock-free queue in the order in which they are generated. The lock-free queue takes out the operation type codes in sequence and converts the operation type codes back to the input and output requests or the internal operations of the data server. The input and output requests or the internal operations of the data server are executed in the order in which they are taken out. The input and output requests or the internal operations of the data server inherit the structure of the same context. Fig.11 This embodiment achieves a global lock-free effect by executing input and output requests on the queue without locks.

[0087] In this embodiment, the above-mentioned data server can avoid the double writing of the standard Raft when writing WAL and append file when applying; the delay can be reduced by orderly replying to out-of-order writing and maintaining order when appending, and the fast node does not have to wait for the slow node; in order to meet the extremely high performance requirements of ESSD, the append write-based majority write protocol can be implemented between the DBS client and the data server. Through the promotion of SCL (segment complete LSN), the commit can be submitted when the majority mechanism quorum is met, providing better concurrency. The majority protocol of DBS is realized through the cooperation between the data-client and the data server.

[0088] This embodiment also provides a device for storing replica data on a disk based on distributed block storage.

[0089] like Fig.12 As shown, the device comprises:

[0090] The writing module 1202 is used to write the replica data block corresponding to the write request to the physical disk when receiving the write request of the replica data block of the target replica, wherein the write request is used to request to write the replica data block to the disk;

[0091] A reply module 1204 is used to return a reply message, wherein the reply message carries the copy length of the target copy that has been written to the physical disk;

[0092] The read module 1206 is used to, upon receiving a read request to read a target replica, return the replica data block requested by the read request if the replica data block requested by the read request is within the replica length of the target replica; and return a prompt message based on the search result of the replica data block requested by the read request if the replica data block requested by the read request is outside the replica length of the target replica.

[0093] The above-mentioned replica data storage device can be applied to the process of storing replica data in elastically scalable distributed block storage (Elastic blockstore, EBS).

[0094] In distributed storage, in order to ensure that the business data on the physical server will not be affected when the physical server crashes, the business data is usually written into multiple copies, each of which is called a Replica copy. Different copies can be written to different servers. This ensures that the business data will not be lost and can be accessed normally when the server crashes. The above-mentioned target copy can be a copy of the business data, and the write to disk means writing the copy data from the server to the physical disk.

[0095] In order to implement the above-mentioned method of storing replica data on disk, a system can be built. The schematic diagram of the system is as follows: Figure 2 As shown in the figure, users perform read and write operations on replicas through the cloud platform and virtual machines at the user layer, generating input / output (IO) requests (read / write requests). The block controller at the distributed block storage logic layer schedules the read / write requests, the block server forwards the read / write requests, the data controller at the storage layer schedules the replica data, and the data server reads and writes the replica data.

[0096] In the above system, the server used for data placement is the data server, which is a server on the data path of the physical disk for single-node data IO request placement and resource management; the block server block-server is a server that calls the data-client interface to initiate IO requests to the data server where all replicas are located. It is responsible for accepting customer IO requests, applying for resources, issuing IO requests and establishing indexes; the block controller block-master is a server that controls the block-server; the data controller data-master is a server on the physical disk path that is responsible for all data server resource management, replica recovery and balancing; the data-client is a client that initiates writing to multiple replicas.

[0097] In the above system, the path diagram of the IO request initiated by the data-client is as follows Figure 3As shown, an IO request is generated from a virtual machine, sent to a block server through a distributed block storage engine, and then forwarded to a data server. The data server then performs a disk write operation (write operation) or a read operation on the replica data according to the IO request.

[0098] For the above data server, in terms of reading and writing, the copy data can be written to the physical disk through the read and write functions. Figure 6 The following is a schematic diagram of the write process of the data-client of the DBS database. The user initiates a request, applies for memory from the memory pool through the spdk thread, then queries the replica location, sends the IO request to the data master, and the data master sends the IO request to the corresponding data server. Figure 6 In the example, the IO request is sent to data server 1, data server 2, and data server 3 respectively, and responses are obtained. Finally, the response is fed back to the user and the memory is released. It should be noted that the order of sending requests 0-2 and the order of obtaining responses 0-2 in the figure are examples and do not constitute a limitation. After the data-client IO request is sent to the data server, the data server can write the target copy requested by the IO request to the physical disk. Figure 7The following is a write flow diagram of the data server of the DBS database. The write request is sent to the data server through the UcxServerMsgHandlerPolling thread, and the data server performs the write operation. Then the spkd thread checks the data block chunk to check whether it exists, whether the status is readable and writable, and whether the data address in the data block is continuous, whether the data exceeds the boundary, and whether it is an overwrite write. Then the request is sent to the user's distributed block system and the response is received, and the chunk is checked again to check whether it exists, whether the status is readable and writable, and whether the IO is continuous. When the data server writes data, it performs out-of-order writing, and after writing, it performs an orderly reply to the user. For the target replica, it can be composed of replica data blocks. When the data server writes the target replica to the physical disk, it writes each replica data block of the target replica to the physical disk. Since there may be a write failure in the replica data block, the data server must reply the replica length of the written target replica after writing the target replica to the physical disk. Replica length: The farthest position where the data server replica has determined that the data is continuously stored on the disk and there is no hole in front, refers to the length of the continuous data written to the physical disk. For example, if the replica data blocks are 1-10, and the data blocks written to the physical disk and effective are 1-6, 8, and 9, then the replica length is 6. If the replica data blocks written to the physical disk are 1-3, 4-8, and 10, then the replica length of the target replica written to the physical disk is 3. The consistency point of the replica length refers to the replica length reached by more than half of the replicas. It is defined as the length of the current replica returned to the data client after the IO of each replica is completed. The length that more than half of the replicas exceed or reach is called the consistency point. For example, if there are 3 replicas, and the replica length of two of them reaches 3, the consistency point is 3. After the dataserver completes writing the target replica, it replies to the block-server with the replica length and consistency point of the target replica. When the client requests to read data, it can initiate an IO request to read data, such as Figure 8 The following is a diagram of the read process of the data-client of the DBS database. The user initiates a request, applies for memory from the memory pool through the spdk thread, and then queries the replica location from the datamaster, finds out which data server the replica is located on, and sends the IO request to the data server. The block-server determines which data servers the client requests to read from through the data-master and determines a data server to send the request to, and the data server provides the data. Fig. 9The following is a read flow diagram of the data-server of the DBS database. The read request is sent to the data server through the UcxServerMsgHandlerPolling thread, and the data server performs the read operation. The spkd thread checks the data block cucnk to see if it exists and whether it is readable and writable. If it exists and is readable and writable, check whether the data address in the data block is continuous and whether the data exceeds the limit. The data server replies the read data to the user. Since the data server returns the replica length after writing the target replica, the data within the replica length requested by the client can be returned to the client, but the block-server does not guarantee that it can return the data outside the replica length requested by the client.

[0099] For other examples of this embodiment, please refer to the above examples and will not be described in detail here.

[0100] like Fig.13 As shown, an embodiment of the present application provides an electronic device, including a processor 111, a communication interface 112, a memory 113 and a communication bus 114, wherein the processor 111, the communication interface 112, and the memory 113 communicate with each other through the communication bus 114.

[0101] A memory 113, for storing computer programs;

[0102] In one embodiment of the present application, the processor 111 is used to execute the program stored in the memory 113 to implement any one of the above method embodiments to provide a method for storing replica data on a disk based on distributed block storage, including:

[0103] When receiving a write request for a replica data block of the target replica, writing the replica data block corresponding to the write request to the physical disk, wherein the write request is used to request that the replica data block be written to the disk;

[0104] Return a reply message, wherein the reply message carries the copy length of the target copy that has been written to the physical disk;

[0105] In case of receiving a read request to read the target replica, if the replica data block requested by the read request is within the replica length of the target replica, returning the replica data block requested by the read request;

[0106] In the case that the replica data block requested by the read request is outside the replica length of the target replica, a prompt message is returned according to the search result of the replica data block requested by the read request.

[0107] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for storing replica data on a disk based on distributed block storage as provided in any of the aforementioned method embodiments are implemented.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0110] It should be understood that the terms used herein are only for the purpose of describing specific example embodiments and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "include", "comprise", "contain", and "have" are inclusive, and therefore specify the existence of stated features, steps, operations, elements and / or parts, but do not exclude the existence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not interpreted as necessarily requiring them to be performed in the specific order described or illustrated, unless the execution order is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0111] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for storing replica data on a disk based on distributed block storage, characterized in that: include: When receiving a write request for a replica data block of a target replica, writing the replica data block corresponding to the write request to a physical disk, wherein the write request is used to request that the replica data block be written to the disk; Returning a reply message, wherein the reply message carries the copy length of the target copy that has been written to the physical disk; In case of receiving a read request to read the target replica, if the replica data block requested by the read request is within the replica length of the target replica, returning the replica data block requested by the read request; In the case that the replica data block requested by the read request is outside the replica length of the target replica, a prompt message is returned according to the search result of the replica data block requested by the read request.

2. The method according to claim 1, characterized in that: When receiving a write request for a replica data block of a target replica, writing the replica data block corresponding to the write request to the physical disk comprises: When receiving the write request, writing the replica data block to the physical disk in a random write order; Starting from the first replica data block of the target replica, the length of a plurality of continuous and uninterrupted replica data blocks is determined as the replica length.

3. The method according to claim 2, characterized in that The step of determining the length of a plurality of continuous and uninterrupted replica data blocks as the replica length starting from the first replica data block of the target replica comprises: When any replica data block is written to the physical disk, the write request corresponding to the replica data block is stored in a message confirmation queue, wherein the write request in the message confirmation queue is a request that has been received and written to the physical disk; In a case where there are 1st to Nth write requests in the message confirmation queue and the N+1th write request is not included, the N is determined as the copy length.

4. The method according to claim 1, characterized in that: In the case where the replica data block requested by the read request is outside the replica length of the target replica, returning a prompt message according to the search result of the replica data block requested by the read request includes: Searching for the replica data block requested by the read request on the target physical disk; When the replica data block requested by the read request is found, returning the replica data block; If the replica data block requested by the read request is not found, searching for the replica data block requested by the read request from other physical disks, wherein the other physical disks are other physical disks where the target replica is written; In case the replica data block requested by the read request is found in the other physical disk, returning the replica data block; If the replica data block requested by the read request is not found in the other physical disk, an error message is returned.

5. The method according to claim 1, characterized in that After writing the target copy to the physical disk, the method further includes: When the copy replenishment operation is triggered, the longest target copy among all target copies of other physical disks is determined as the data source; Setting the data source to a write-prohibited state; The data source is used to perform a copy-repair operation on the target copy on the target physical disk.

6. The method according to claim 5, characterized in that The using the data source to perform a copy-repair operation on the target copy on the target physical disk comprises: Reading data from an offset position of the data source, wherein the offset position is a position in the data source that is offset by a copy length of the target copy; Writing the read data into the target copy until the target copy has the same length as the data source; The replica length of the target replica is updated.

7. The method according to claim 1, characterized in that In case a write request for a replica data block of a target replica is received, the method further comprises: assigning a first operation type code to the write request; Inserting the first operation type code into a lock-free queue, wherein the lock-free queue also includes a second operation type code, and the second operation type code is a code for internal operations of a data server for storing data on a disk; The write request and the internal operation are executed in the order of the operation type encoding in the lock-free queue.

8. A device for storing replica data on a disk based on distributed block storage, characterized in that: include: A writing module, configured to, upon receiving a write request for a replica data block of a target replica, write the replica data block corresponding to the write request to a physical disk, wherein the write request is used to request that the replica data block be written to the disk; A reply module, used for returning a reply message, wherein the reply message carries the copy length of the target copy that has been written to the physical disk; A reading module is used to, upon receiving a read request to read the target replica, return the replica data block requested by the read request if the replica data block requested by the read request is within the replica length of the target replica; and to return a prompt message based on the search result of the replica data block requested by the read request if the replica data block requested by the read request is outside the replica length of the target replica.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images