A method and system for distributed storage multi-client asynchronous writing

Abstract

The present invention provides a method and system for asynchronous writing of multiple clients in distributed storage, the method comprising receiving a write operation request from a client, parsing the offset and length of the current write, and using the version number assigned to the current request as a key value, the offset and length as a value, and saving the key-value pair; for the write operation of the same file, checking the overlap degree of the values ​​in the key-value pair; performing filtering and integration; the client writes the data into the storage system and searches for the corresponding version number, if the current version number does not exist, the data is discarded, otherwise the data is written to the disk according to the value corresponding to the current version number. When multiple clients write a file at the same time, the present invention adopts an asynchronous write mode, and assigns a version number to the write data, and checks the offset and length of different write operations based on the version number, integrates the version numbers, avoids data redundancy, and makes full use of the client's local cache to perform cache write operations, thereby improving the client's write efficiency.

Description

Technical Field

[0001] The present invention relates to the field of storage technology, and in particular to a method and system for distributed storage multi-client asynchronous writing. Background Art

[0002] In a distributed storage system, when multiple clients write the same file, in order to ensure data consistency, the client local cache is bypassed and the data is written to the storage system in a synchronous write manner.

[0003] Therefore, when multiple clients write the same file, the local cache is bypassed, resulting in a higher write latency than the write cache, resulting in lower write efficiency. Summary of the invention

[0004] The present invention provides a distributed storage multi-client asynchronous writing method and system, which are used to solve the problem of low efficiency of existing synchronous writing.

[0005] To achieve the above object, the present invention adopts the following technical solutions:

[0006] A first aspect of the present invention provides a method for distributed storage multi-client asynchronous writing, the method comprising the following steps:

[0007] Receive a write operation request from the client, parse the offset and length of the current write, and save the key-value pair using the version number assigned to the current request as the key value and the offset and length as the value;

[0008] For a write operation on the same file, checking the overlap degree of the values ​​in the key-value pairs;

[0009] When the values ​​completely overlap, version number filtering is performed to retain the latest version number;

[0010] When the values ​​partially overlap, the version numbers are integrated, the latest version number is retained, the old version number is deleted, or the value corresponding to the old version number is modified;

[0011] The client writes the data to the storage system and searches for the corresponding version number. If the current version number does not exist, the data is discarded. Otherwise, the data is written to the disk according to the value corresponding to the current version number.

[0012] Furthermore, the method further comprises the steps of:

[0013] The client requests the server for the version number corresponding to this write operation, and sends the offset and length of this write operation to the server.

[0014] Furthermore, the method further comprises the following steps:

[0015] After receiving the version number assigned by the server, the client writes the data of the current write operation into the local cache.

[0016] Furthermore, the partial overlap of the values ​​includes that the offset of the subsequent write operation is smaller than the offset of the previous write operation, and the data end point of the subsequent write operation is within the data end point of the previous write operation. The version number integration is specifically as follows:

[0017] The offset of the previous write operation is modified to offset2+length2-offset1, where offset2, length2, and offset1 represent the offset of the next write operation, the length of the next write operation, and the offset of the previous write operation, respectively.

[0018] Furthermore, the partial overlap of the values ​​includes that the offset of the subsequent write operation is greater than the offset of the previous write operation, and the data end point of the previous write operation is within the subsequent write operation, and the version number integration is specifically as follows:

[0019] The length of the previous write operation is modified to offset2-offset1, where offset2 and offset1 represent the offset of the subsequent write operation and the offset of the previous write operation respectively.

[0020] Furthermore, the partial overlap of the values ​​includes that the offset of the subsequent write operation is smaller than the offset of the previous write operation, and the end point of the previous write operation is within the subsequent write operation, and the version number integration is specifically as follows:

[0021] Delete the version number corresponding to the previous write operation.

[0022] Furthermore, the partial overlap of the values ​​includes that the offset of the subsequent write operation is greater than the offset of the previous write operation, and the end point of the subsequent write operation is within the previous write operation, and the version number integration is specifically as follows:

[0023] The value of the previous write operation is divided into two, specifically including a first value (offset1, length1') and a second value (offset1', length1"); wherein offset1 is the offset of the previous write operation, length1' is the modified first length, satisfying length1'=offset2-offset1; offset1' is the modified offset, satisfying offset1'=offset2+length2; length1" is the modified second length, satisfying length1"=length1+offset1-offset1'.

[0024] A second aspect of the present invention provides a distributed storage multi-client asynchronous writing system, including a server and multiple clients, wherein the server includes:

[0025] A request processing unit, used to receive a write operation request from a client, parse the offset and length of the current write, and save the key-value pair using the version number assigned to the current request as a key value and the offset and length as a value;

[0026] The version processing unit checks the overlap degree of the values ​​in the key-value pairs for the write operation of the same file; when the values ​​completely overlap, performs version number filtering and retains the latest version number; when the values ​​partially overlap, performs version number integration, retains the latest version number, deletes the old version number or modifies the value corresponding to the old version number;

[0027] The client includes a data writing unit, which writes data into the cache system and searches for the corresponding version number. If the current version number does not exist, the data is discarded; otherwise, the data is written to the disk according to the value corresponding to the current version number.

[0028] Furthermore, the client further includes a version number request unit, and the server further includes a version number allocation unit;

[0029] The version number request unit is used to request the version number corresponding to the current write operation from the server, and send the offset and length of the current write operation to the server;

[0030] The version number allocation unit allocates a version number to the client based on the version number request of the client, and the version numbers are incremented sequentially.

[0031] Furthermore, the client also includes a cache unit, which writes data into a local cache after receiving the version number. The control device of the network service in the second aspect of the present invention can implement the method in the first aspect and each implementation method of the first aspect, and achieve the same effect.

[0032] The effects provided in the content of the invention are only the effects of the embodiments, not all the effects of the invention. One of the above technical solutions has the following advantages or beneficial effects:

[0033] When multiple clients write a file at the same time, the present invention adopts an asynchronous writing method, assigns version numbers to the written data, checks the offsets and lengths of different write operations based on the version numbers, integrates the version numbers to avoid data redundancy, and makes full use of the client's local cache to perform cache write operations, thereby improving the client's write efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0035] Figure 1 It is a schematic diagram of the process of the method of the present invention;

[0036] Figure 2 This is a schematic diagram of the partial overlap of the values ​​in the key-value pairs of the present invention.

[0037] Figure 3 It is a structural schematic diagram of the system of the present invention. DETAILED DESCRIPTION

[0038] In order to clearly illustrate the technical features of the present solution, the present invention is described in detail below through specific implementation methods and in conjunction with the accompanying drawings. The disclosure below provides many different embodiments or examples for realizing different structures of the present invention. In order to simplify the disclosure of the present invention, the components and settings of specific examples are described below. In addition, the present invention may repeat reference numbers and / or letters in different examples. This repetition is for the purpose of simplification and clarity, and does not itself indicate the relationship between the various embodiments and / or settings discussed. It should be noted that the components illustrated in the accompanying drawings are not necessarily drawn to scale. The present invention omits the description of known components and processing techniques and processes to avoid unnecessary limitations on the present invention.

[0039] like Figure 1 As shown, a distributed storage multi-client asynchronous writing method according to an embodiment of the present invention includes the following steps:

[0040] S1, receiving a write operation request from the client, parsing the offset and length of the current write, and using the version number assigned to the current request as the key value, the offset and length as the value, and saving the key-value pair;

[0041] S2, for the write operation of the same file, checking the overlap degree of the values ​​in the key-value pairs;

[0042] S31, when the values ​​completely overlap, version number filtering is performed to retain the latest version number;

[0043] S32, when the values ​​partially overlap, perform version number integration, retain the latest version number, delete the old version number or modify the value corresponding to the old version number;

[0044] S4, the client writes the data to the storage system and searches for the corresponding version number. If the current version number does not exist, the data is discarded. Otherwise, the data is written to the disk according to the value corresponding to the current version number.

[0045] The method further comprises the steps of:

[0046] The client requests the server for the version number version corresponding to this write operation, and sends the offset offset and length length of this write operation to the server.

[0047] In step S1, after receiving the write operation request from the client, the server parses the client's (offset, length), assigns a version to it and returns it to the client, and saves the map with version as the key value and (offset, length) as the value.

[0048] After receiving the version number version assigned by the server, the client writes the data of the current write operation into the local cache.

[0049] In step S2, when performing a write operation on the same file, the server automatically performs a filtering and integration operation on the map.

[0050] In step S31, filtering means that for completely repeated (offset, version), the latest version is retained and other versions are deleted from the map.

[0051] In step S32, Figure 2 As shown, for partially overlapping (offset, length), integration is performed.

[0052] For the first case, the offset of the subsequent write operation is smaller than the offset of the previous write operation, and the data endpoint of the subsequent write operation is within the previous write operation. The version number integration is specifically as follows: modify the offset of the previous write operation to offset1'=offset2+length2-offset1, where offset2, length2 and offset1 represent the offset of the subsequent write operation, the length of the subsequent write operation and the offset of the previous write operation, respectively.

[0053] For the second case, the offset of the subsequent write operation is greater than the offset of the previous write operation, and the data endpoint of the previous write operation is within the subsequent write operation. The version number integration is specifically as follows: modify the length of the previous write operation to length'=offset2-offset1, where offset2 and offset1 represent the offset of the subsequent write operation and the offset of the previous write operation, respectively.

[0054] For the third case, the offset of the subsequent write operation is smaller than the offset of the previous write operation, and the end point of the previous write operation is within the subsequent write operation, and the version number integration is specifically: deleting the version number corresponding to the previous write operation.

[0055] For the fourth case, the offset of the subsequent write operation is greater than the offset of the previous write operation, and the end point of the subsequent write operation is within the previous write operation. The version number integration is specifically as follows: the value of the previous write operation is divided into two, specifically including a first value (offset1, length1') and a second value (offset1', length1"); wherein offset1 is the offset of the previous write operation, length1' is the modified first length, satisfying length1'=offset2-offset1; offset1' is the modified offset, satisfying offset1'=offset2+length2; length1" is the modified second length, satisfying length1"=length1+offset1-offset1'.

[0056] like Figure 3 As shown, an embodiment of the present invention provides a distributed storage multi-client asynchronous writing system, including a client 1 and multiple clients 2, the server 2 includes a version number allocation unit 21, a request processing unit 22 and a version processing unit 23; the client 1 includes a version number request unit 11, a cache unit 12 and a data writing unit 13.

[0057] The version number request unit 11 is used to request the version number corresponding to the current write operation from the server, and send the offset and length of the current write operation to the server at the same time; the version number allocation unit 21 allocates a version number to the client based on the client's version number request, and the version number is sequentially increased; after receiving the version number, the cache unit 12 writes the data to the local cache; the request processing unit 21 receives the client's write operation request, parses the offset and length of the current write, and uses the version number allocated to the current request as the key value, and the offset and length as the value to save the key-value pair; the version processing unit 23 checks the overlap degree of the values ​​in the key-value pair for the write operation of the same file; when the values ​​completely overlap, the version number is filtered to retain the latest version number; when the values ​​partially overlap, the version number is integrated to retain the latest version number, delete the old version number or modify the value corresponding to the old version number; the data writing unit 13 writes the data into the cache system and searches for the corresponding version number. If the current version number does not exist, the data is discarded, otherwise the data is written to the disk according to the value corresponding to the current version number.

[0058] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A distributed storage multi-client asynchronous writing method, characterized in that: The method comprises the following steps: Receive a write operation request from the client, parse the offset and length of the current write, and save the key-value pair using the version number assigned to the current request as the key value and the offset and length as the value; For a write operation on the same file, checking the overlap degree of the values ​​in the key-value pairs; When the values ​​completely overlap, version number filtering is performed to retain the latest version number; When the values ​​partially overlap, the version numbers are integrated, the latest version number is retained, the old version number is deleted, or the value corresponding to the old version number is modified; The client writes the data to the storage system and searches for the corresponding version number. If the current version number does not exist, the data is discarded. Otherwise, the data is written to the disk according to the value corresponding to the current version number. The partial overlap of the values ​​includes that the offset of the subsequent write operation is smaller than the offset of the previous write operation, and the data end point of the subsequent write operation is within the previous write operation. The version number integration is specifically as follows: The offset of the previous write operation is modified to offset2+length2-offset1, where offset2, length2, and offset1 represent the offset of the next write operation, the length of the next write operation, and the offset of the previous write operation, respectively.

2. According to claim 1, the distributed storage multi-client asynchronous writing method is characterized in that: The method further comprises the steps of: The client requests the server for the version number corresponding to this write operation, and sends the offset and length of this write operation to the server.

3. The method for distributed storage multi-client asynchronous writing according to claim 2 is characterized in that: The method further comprises the following steps: After receiving the version number assigned by the server, the client writes the data of the current write operation into the local cache.

4. The method for distributed storage multi-client asynchronous writing according to any one of claims 1 to 3, characterized in that: The partial overlap of the values ​​includes that the offset of the subsequent write operation is greater than the offset of the previous write operation, and the data end point of the previous write operation is within the data end point of the subsequent write operation. The version number integration is specifically as follows: The length of the previous write operation is modified to offset2-offset1, where offset2 and offset1 represent the offset of the subsequent write operation and the offset of the previous write operation respectively.

5. The method for distributed storage multi-client asynchronous writing according to any one of claims 1 to 3, characterized in that: The partial overlap of the values ​​includes that the offset of the subsequent write operation is smaller than the offset of the previous write operation, and the end point of the previous write operation is within the subsequent write operation. The version number integration is specifically as follows: Delete the version number corresponding to the previous write operation.

6. The distributed storage multi-client asynchronous writing method according to any one of claims 1 to 3, characterized in that: The partial overlap of the values ​​includes that the offset of the subsequent write operation is greater than the offset of the previous write operation, and the end point of the subsequent write operation is within the previous write operation. The version number integration is specifically as follows: The value of the previous write operation is divided into two, specifically including a first value (offset1, length1') and a second value (offset1', length1"); wherein offset1 is the offset of the previous write operation, length1' is the modified first length, satisfying length1'=offset2-offset1; offset1' is the modified offset, satisfying offset1'=offset2+length2; length1" is the modified second length, satisfying length1"=length1+offset1-offset1'.

7. A distributed storage multi-client asynchronous writing system, comprising a server and multiple clients, characterized in that: The system is used to implement the method according to claim 1, and the server includes: A request processing unit, used to receive a write operation request from a client, parse the offset and length of the current write, and save the key-value pair using the version number assigned to the current request as a key value and the offset and length as a value; The version processing unit checks the overlap degree of the values ​​in the key-value pairs for the write operation of the same file; when the values ​​completely overlap, performs version number filtering and retains the latest version number; when the values ​​partially overlap, performs version number integration, retains the latest version number, deletes the old version number or modifies the value corresponding to the old version number; The client includes a data writing unit, which writes data into the cache system and searches for the corresponding version number. If the current version number does not exist, the data is discarded; otherwise, the data is written to the disk according to the value corresponding to the current version number.

8. The distributed storage multi-client asynchronous writing system according to claim 7 is characterized in that: The client also includes a version number request unit, and the server also includes a version number allocation unit; The version number request unit is used to request the version number corresponding to the current write operation from the server, and send the offset and length of the current write operation to the server; The version number allocation unit allocates a version number to the client based on the version number request of the client, and the version numbers are incremented sequentially.

9. The distributed storage multi-client asynchronous writing system according to claim 8 is characterized in that: The client further includes a cache unit, which writes the data into a local cache after receiving the version number.

Images