Object storage batch reading and writing methods, clients, storage gateways and media

Abstract

The embodiment of the present invention relates to a method for batch reading and writing of object storage, a client and a storage gateway. The reading method is applied to the client and the storage gateway, and the reading method enables the client to read the shard data of a specified range of at least one specified object from the storage gateway, thereby realizing batch reading and avoiding multiple interactions between the client and the storage gateway; the writing method is applied to the client and the storage gateway, and the writing method can realize the full or partial writing of at least one object and shard data, realize the writing of massive objects, and effectively reduce the number of interactions between the client and the storage gateway. The embodiment of the present invention reduces the number of request interactions between the client and the storage gateway, improves the reading and writing efficiency, and reduces the client resource consumption by providing a batch reading and writing interface based on POST requests.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of object storage, and in particular to an object storage batch reading and writing method, a client, a storage gateway and a medium. Background Art

[0002] Object storage was first introduced by Amazon in 2006 as S3 (Simple Storage Service). Object storage is object-oriented / file-oriented, massive Internet storage, which can also be directly called "cloud storage". Although objects are files, they are encapsulated files (objects in programming have the characteristic of encapsulation), that is, in an object storage system, files cannot be directly opened / modified, but files can be uploaded and downloaded like FTP.

[0003] Taking distributed object storage as an example, the implementation of http-type application clients is based on the S3 REST API standard when reading and writing. However, for existing S3-type storage interfaces, whether it is AWS or public cloud products, if you want to write multiple files, you can only write one file at a time. The client needs to encapsulate PUT write requests for multiple files separately, and then send the requests at a certain concurrency. The interaction consumes a lot of client resources and network resources. When reading files, sometimes you only need to read part of the file's fragments, that is, part of the range of data. The usual practice is to use GET read requests to specify multiple different ranges, and you may need to encapsulate GET read requests for different fragments separately, and then send the requests at a certain concurrency. In particular, when performing GET reading on fragments of different files, if there are a lot of files, you have to construct a lot of requests and interact a lot, which also consumes the overall system resources and is inefficient. Summary of the invention

[0004] The purpose of the embodiment of the present invention is to provide an object storage batch reading and writing method, a client, a storage gateway and a storage medium. By providing a batch reading and writing interface based on POST requests, the number of request interactions between the client and the storage gateway is reduced, the reading and writing efficiency is improved, and the client resource consumption is reduced.

[0005] In a first aspect, an embodiment of the present invention provides an object storage batch reading method, which is applied to a client; the method includes:

[0006] When receiving a read operation instruction for at least one specified object corresponding to the target bucket information, encapsulate a read POST request; the read POST request includes a read request header, target bucket information located in the read request header or the read request line, and a shard range to be read located in the read request body;

[0007] Sending the read POST request to the storage gateway, so that the storage gateway responds according to the read POST request after receiving the read POST request;

[0008] Read multiple shard data of the shard range to be read of the specified object according to the response of the storage gateway.

[0009] In some embodiments, the encapsulating reading the POST request includes:

[0010] In the read request line, determine the request method as read;

[0011] Put the target bucket information into the read request header;

[0012] A multi-level tag is added to the body of the read request body to identify the specified object and the fragmented data of the specified range of the specified object, wherein the number of the specified object is 1.

[0013] In some embodiments, the encapsulation reads the POST request and further includes:

[0014] The read POST request is encapsulated according to the API interface specification defined by the storage, wherein the API interface is based on the S3 REST API protocol standard.

[0015] In some embodiments, if the number of the specified objects is greater than 1, adding a multi-level tag in the body of the request body to identify the specified objects and the fragmented data of the specified range of the specified objects includes:

[0016] Multi-level tags are added to the body of the read request body, and different identifiers are added to different designated objects to identify the designated objects and the fragmented data of the designated range of the designated objects.

[0017] In some embodiments, the method further comprises:

[0018] If the read POST request contains at least two specified objects, the number of the specified objects is at most 200, and the number of shard data of each specified object is at most 200; if the read POST request contains only one specified object, the number of shard data of the specified object is at most 1000.

[0019] In a second aspect, an embodiment of the present invention provides an object storage batch writing method, which is applied to a client; the method includes:

[0020] Create a write POST request, wherein the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body, and shard data of the objects to be processed;

[0021] Sending the write POST request to the storage gateway, so that after receiving the write POST request, the storage gateway parses the write request header to request a response to the target bucket information, the object to be processed, and the shard data, obtains a response result, writes the target bucket information, the object to be processed, and the shard data according to the response result, and returns the response result to the client;

[0022] After receiving the response result, if there are objects to be processed whose request is passed in the response result, the processing result of each object to be processed whose request is passed in the response result is parsed.

[0023] In some embodiments, the creating a write POST request, the writing POST request including a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body and shard data of the objects to be processed, includes:

[0024] In the write request line, determine the request method as write;

[0025] Put the target bucket information into a write request header or a write request line;

[0026] Setting an object dividing line in the write request header;

[0027] In the body of the request, the object separator line is used to start the request.

[0028] Encapsulate the first object to be processed, wherein the object to be processed includes the object name, the fragmentation data of the object to be processed, and additional attributes;

[0029] Continue to encapsulate the second object to be processed, and separate the first object to be processed from the second object to be processed by the object dividing line until all objects to be processed are encapsulated;

[0030] Add an end dividing line to complete the encapsulation of the write POST request.

[0031] In some embodiments, the creating and writing POST request further includes:

[0032] The write POST request is encapsulated according to the API interface specification defined by the storage, wherein the API interface is based on the S3 REST API protocol standard.

[0033] In some embodiments, the method further comprises:

[0034] If there is an object to be processed whose request has not been passed in the response result, receiving an error code of the object to be processed whose request has not been passed sent by the storage gateway;

[0035] The failure reason is determined according to the error code, and the failed object to be processed is corrected according to the failure reason, so as to re-create a write POST request according to the corrected object to be processed.

[0036] In a third aspect, an embodiment of the present invention further provides an object storage batch reading method, which is applied to a storage gateway, and the method includes:

[0037] After receiving the read POST request sent by the client, respond according to the read POST request, wherein the read POST request includes a read request header, target bucket information located in the read request header or request line, at least one specified object located in the read request body and shard data of the specified object, and the specified object is the object to be processed that passes the request.

[0038] In some embodiments, responding according to the read POST request includes:

[0039] Performing a validity check on the read POST request;

[0040] When the read POST request passes the legality check, parsing the body of the read request body to obtain the specified object and the specified range of the specified object;

[0041] Searching for a location corresponding to the specified range in the stored shard data;

[0042] Integrate and / or sort the storage order of the stored shard data according to the positions to obtain a reading order;

[0043] Read the shard data in the specified range according to the reading order;

[0044] The fragment data in the specified range is encapsulated in a response result, and the response result is sent to the client.

[0045] In some embodiments, the number of the specified objects is 1, and encapsulating the shard data of the specified range in the response result includes:

[0046] Put the response result in the response request line;

[0047] Set the fragment separation line in the response request header;

[0048] The body of the response request begins with a fragment separator line;

[0049] Encapsulate the first fragment data of the specified range of the specified object, the first fragment data including the fragment range and data;

[0050] Encapsulate the second fragment data in the specified range until all fragment data in the specified range are encapsulated, and adjacent fragment data are separated by fragment dividing lines;

[0051] Add a slice separator line to end the encapsulation of the slice data in the specified range.

[0052] In some embodiments, if the number of the specified objects is greater than 1, the shard data of the specified range is encapsulated in the response result, including:

[0053] Put the response result in the response request line;

[0054] Set the fragment separation line in the response request header;

[0055] The body of the response request begins with a fragment separator line;

[0056] Encapsulate the shard data of the specified range of the first specified object, and the shard data of the specified range of the first specified object are separated by shard dividing lines;

[0057] Encapsulate the fragmented data of the specified range of the second specified object until encapsulating the fragmented data of the specified range of all the specified objects; and the fragmented data of the specified range of the second specified object are separated by the fragmentation dividing line;

[0058] Add an object separator line to end the encapsulation of the slice data of the specified range of all specified objects.

[0059] In a fourth aspect, an embodiment of the present invention further provides an object storage batch writing method, which is applied to a storage gateway, and the method includes:

[0060] Receive a write POST request sent by a client; the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body, and shard data of the objects to be processed;

[0061] Parse the write request header to request a response to the target bucket information, the object to be processed, and the shard data, obtain a response result, write the target bucket information, the object to be processed, and the shard data according to the response result, and return the response result to the client.

[0062] In some embodiments, parsing the write request header to request a response to the target bucket information, the object to be processed, and the shard data, obtaining a response result, writing the target bucket information, the object to be processed, and the shard data according to the response result, and returning the response result to the client includes:

[0063] Parse the write request header, perform a legality check on the write POST request, and obtain a response result corresponding to the object to be processed;

[0064] Placing the response result in a response request line;

[0065] Setting an object dividing line, and placing the object dividing line in the response request header;

[0066] The body of the response request begins with the response dividing line;

[0067] Encapsulate the response result of the first object, the response result including the object name, response code and additional attributes;

[0068] Encapsulate the response result of the second object, and separate the response result of the first object from the response result of the second object by a response dividing line, until the encapsulation of the response results of all objects is completed;

[0069] Add an end dividing line to complete the encapsulation of the response result.

[0070] In some embodiments, the performing of a validity check on the write POST request to obtain a response result corresponding to the object to be processed includes:

[0071] Verify the legitimacy of the request interface for writing the POST request, wherein, when the legitimacy verification of the request interface for writing the POST request fails, obtain a first response result that the legitimacy verification of the request interface for writing the POST request fails;

[0072] When the legitimacy check of the request interface for writing the POST request passes, a second response result indicating that the legitimacy check of the request interface for writing the POST request passes is obtained, and the target bucket information and the permissions of the object to be processed are verified;

[0073] When the target bucket information and the permission check of the specified object are passed, a third response result indicating that the check is passed is obtained, and the target bucket information, the object to be processed, and the shard data are stored;

[0074] When the permission check of the target bucket information and the object to be processed fails, a fourth response result indicating that the check failed is obtained.

[0075] In a fifth aspect, an embodiment of the present invention provides a client, the client comprising:

[0076] at least one first processor, and

[0077] A first memory, wherein the first memory is communicatively connected to the first processor, and the first memory stores instructions executable by the at least one first processor, wherein the instructions are executed by the at least one first processor so that the at least one processor can execute any of the methods described above.

[0078] In a sixth aspect, an embodiment of the present invention provides a storage gateway, the storage gateway comprising:

[0079] at least one second processor, and

[0080] A second memory, the second memory is communicatively connected to the second processor, the second memory stores instructions executable by the at least one second processor, the instructions are executed by the at least one second processor so that the at least one processor can execute any of the methods described above.

[0081] In the seventh aspect, an embodiment of the present invention provides a non-volatile computer-readable and writable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a client or a storage gateway, the client executes the method described above or the storage gateway executes the method described above.

[0082] The object storage batch reading and writing method, client, storage gateway and storage medium of the embodiment of the present invention can write at least one object when writing. Therefore, when the client reads the shard data, similarly, a specified object or multiple specified objects can be read from the storage gateway, and a read POST request can be sent to the storage gateway so that the storage gateway responds according to the read POST request. According to the response of the storage gateway, multiple shard data of the specified object are read, thereby realizing batch reading of multiple shard data.

[0083] Furthermore, when the client writes a batch of objects to the storage gateway, a write POST request is created, and the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body, and the shard data of the objects to be processed, so that one target bucket information corresponds to at least two objects to be processed and their shard data, that is, one write POST request includes multiple objects to be processed and their shard data; the write POST request is sent to the storage gateway, and the storage gateway parses the write request header to request a response to the target bucket information, the objects to be processed, and the shard data, and obtains a response result, writes the target bucket information, the objects to be processed, and the shard data according to the response result, and returns the response result to the client, writes the objects to be processed and the shard data according to the response result, thereby realizing full or partial writing of the objects to be processed and the shard data. This application can effectively reduce the number of interactions, improve access efficiency, and reduce client resource consumption when it is necessary to write massive objects and read shard data of massive objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0085] Figure 1 It is a flowchart of an embodiment of the object storage batch writing method of the present invention;

[0086] Figure 2 It is an interactive schematic diagram of an embodiment of the object storage batch writing and reading method of the present invention;

[0087] Figure 3 It is an interactive schematic diagram of an embodiment of the object storage batch writing method of the present invention;

[0088] Figure 4 It is a schematic diagram of the process of an embodiment of the object storage batch reading method of the present invention;

[0089] Figure 5 is an interactive schematic diagram of reading an object in an embodiment of the object storage batch reading method of the present invention;

[0090] Figure 6 It is a flowchart of reading multiple objects in one embodiment of the object storage batch reading method of the present invention;

[0091] Figure 7 It is a flowchart of an embodiment of the object storage batch writing method of the present invention;

[0092] Figure 8 It is a flowchart of another embodiment of the object storage batch writing method of the present invention;

[0093] Fig. 9 It is a structural schematic diagram of an embodiment of a first object storage batch writing device of the present invention;

[0094] Fig.10 It is a structural schematic diagram of an embodiment of a first object storage batch reading device of the present invention;

[0095] Fig.11 It is a structural schematic diagram of an embodiment of a second object storage batch writing device of the present invention;

[0096] Fig.12 It is a structural schematic diagram of an embodiment of a second object storage batch reading device of the present invention;

[0097] Fig.13 is a schematic diagram of the hardware structure of a first controller in an embodiment of a client of the present invention;

[0098] Fig.14 It is a schematic diagram of the hardware structure of the second controller in an embodiment of the storage gateway of the present invention. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0100] The object storage batch reading / writing method provided in the embodiment of the present invention can be applied to a client, wherein the client can be an intelligent terminal device, such as a smart phone, a computer, an iPad, or other terminal devices.

[0101] The object storage batch reading / writing method provided in the embodiment of the present invention can be applied to a storage gateway, and the storage gateway can be a device such as a server.

[0102] It is understandable that the client is provided with a first controller as the main control center of the client, and the storage gateway is provided with a second controller as the main control center of the storage gateway. By providing a batch read and write interface based on POST requests, the number of request interactions between the client and the storage gateway is reduced, the read and write efficiency is improved, and the client resource consumption is reduced.

[0103] See also Figure 1, Figure 1 The flowchart of the object storage batch writing method provided by the embodiment of the present invention is as follows. The method can be executed by the first controller 13 in the client. Figure 1 and Figure 2 As shown, the method is applied to a client, and the method includes:

[0104] 101: Create a write POST request, wherein the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body, and shard data of the objects to be processed.

[0105] Specifically, writing a POST request refers to an HTTP request message whose request method is POST. The request message includes a request line, a request header, and a request body. The request line includes a request method, URL, and protocol; the request header includes header field names and values, and the target bucket information can be placed in the request line or the request header; the request body includes request data, i.e., at least two objects to be processed and shard data of the objects to be processed.

[0106] The target bucket information can be understood as a bucket (storage space). In the bucket, there are all objects, and each object has multiple shard data. In addition, an object is a file in the client business. Object storage generally calls the stored files objects. Therefore, in this case, the object is the file.

[0107] It can be understood that the target bucket information in this case refers to a bucket, which contains at least two objects to be processed. Each object has a unique file name. Each object can also have separate metadata (additional attributes) and shard data. When a write POST request is constructed, 200 or more objects to be processed can be written to the storage gateway at the same time.

[0108] In some embodiments, creating a write POST request may include:

[0109] The write POST request is encapsulated according to the API interface specification defined by the storage, wherein the API interface is based on the S3 REST API protocol standard.

[0110] Specifically, S3 is Amazon S3, the full name is Amazon Simple Storage Service, REST, is a web service architecture. In simple terms, resources are identified and located by URIs, and operations on resources are defined by HTTP methods. HTTP methods include GET, PUT, POST, DELETE, and HEAD, which constitute actions in REST. These HTTP methods usually match the following CRUD actions:

[0111] Create: POST;

[0112] Read: GET;

[0113] Update: PUT / PATCH;

[0114] Delete: DELETE.

[0115] However, the above matching is not strictly limited. Sometimes PUT can be used to create new resources and POST can be used to update resources. The non-idempotence of POST requests makes it a very flexible method.

[0116] The most critical thing in writing a POST request is how to pass multiple file data together. In order to improve efficiency, a write POST request is limited to batch writing files in the same bucket, so there is only one bucket name, which is directly placed in the request header or request line, and then the file name and shard data of the object to be processed are placed in the request body. In order to separate different file data, a "separator" is also defined in the request header. The request body starts with "--separator" and ends with "--separator--". Different files are separated by the separator. When storing and processing data, different objects will be identified according to the separator (similarly, the separator is used to separate different file response information in the response). After generating the request, the client sends the request to the storage gateway.

[0117] In some embodiments, the creating a write POST request, the writing POST request including a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body and shard data of the objects to be processed, may include:

[0118] In the write request line, determine the request method as write;

[0119] Put the target bucket information into a write request header or a write request line;

[0120] Setting an object dividing line in the write request header;

[0121] In the body of the request, the object separator line is used to start the request.

[0122] Encapsulate the first object to be processed, wherein the object to be processed includes the object name, the fragmentation data of the object to be processed, and additional attributes;

[0123] Continue to encapsulate the second object to be processed, and separate the first object to be processed from the second object to be processed by the object dividing line until all objects to be processed are encapsulated;

[0124] Add an end dividing line to complete the encapsulation of the write POST request.

[0125] Specifically, Figure 3 As shown, the process of constructing a POST request by the client is as follows:

[0126] 1. Fill in put in the URL of the request line to indicate write;

[0127] 2. Put the target bucket (bucketname) in the request header;

[0128] 3. Set the object boundary in the request header;

[0129] 4. Add a blank line;

[0130] 5. The request body starts with a "-- separator line";

[0131] 6. Encapsulate the first object: object name (Key1), additional attributes (Additional), file content (Content);

[0132] 7. Adding "--dividing line" indicates the existence of the next object;

[0133] 8. Encapsulate the second object: object name (Key2), additional attributes (Additional), file content (Content);

[0134] 9. Encapsulate other objects in turn;

[0135] 10. Add “--dividing line--” to indicate the end, there are no other objects.

[0136] By encapsulating multiple objects to be processed into a write POST request and sending them together to the gateway storage, the storage responds after processing the entire write POST request, thereby achieving the purpose of batch writing multiple objects.

[0137] 102: Send the write POST request to the storage gateway so that the storage gateway can parse the write request header to respond to the target bucket information, the object to be processed and the shard data, obtain a response result, write the target bucket information, the object to be processed and the shard data according to the response result, and return the response result to the client.

[0138] After creating a write POST request, the client sends the write POST request to the storage gateway. After receiving the write POST request, the storage gateway parses the write request header to request a response to the target bucket information, the object to be processed, and the shard data, obtains a response result, writes the target bucket information, the object to be processed, and the shard data according to the response result, and returns the response result to the client.

[0139] In some implementation modes, parsing the write request header to request a response to the target bucket information, the object to be processed, and the shard data, obtaining a response result, writing the target bucket information, the object to be processed, and the shard data according to the response result, and returning the response result to the client includes:

[0140] Parse the write request header, perform a legality check on the write POST request, and obtain a response result corresponding to the object to be processed;

[0141] Placing the response result in a response request line;

[0142] Setting an object dividing line, and placing the object dividing line in the response request header;

[0143] The body of the response request begins with the response dividing line;

[0144] Encapsulate the response result of the first object, the response result including the object name, response code and additional attributes;

[0145] Encapsulate the response result of the second object, and separate the response result of the first object from the response result of the second object by a response dividing line, until the encapsulation of the response results of all objects is completed;

[0146] Add an end dividing line to complete the encapsulation of the response result.

[0147] Specifically, Figure 3 As shown in the figure, the encapsulation process of the storage gateway responding to the client's write POST request is as follows:

[0148] 1. Put the request response result in the request line;

[0149] 2. Set the object boundary in the request header;

[0150] 3. Add a blank line;

[0151] 4. The body of the response begins with a “--dividing line”;

[0152] 5. Encapsulate the first object and write a response: object name (Key), error code (Result), additional attributes (Additional);

[0153] 6. Adding a "-- dividing line" indicates that there is a next response;

[0154] 7. Encapsulate the second object and write a response: object name (Key), error code (Result), additional attributes (Additional);

[0155] 8. Encapsulate other objects in turn;

[0156] 9. Add “--dividing line--” to indicate the end, with no other response.

[0157] In addition, the storage gateway performs verification when responding to the request. In some embodiments, the storage gateway performs a validity verification on the write POST request, obtains a response result corresponding to the object to be processed, and stores the target bucket information, at least two objects to be processed, and shard data, which may include:

[0158] Verify the legitimacy of the request interface for writing the POST request, wherein, when the legitimacy verification of the request interface for writing the POST request fails, obtain a first response result that the legitimacy verification of the request interface for writing the POST request fails;

[0159] When the legitimacy check of the request interface for writing the POST request passes, a second response result indicating that the legitimacy check of the request interface for writing the POST request passes is obtained, and the target bucket information and the permissions of the object to be processed are verified;

[0160] When the permission check of the target bucket information and the object to be processed is passed, a third response result indicating that the check is passed is obtained, and the target bucket information, the object to be processed and the shard data are stored;

[0161] When the permission check of the target bucket information and the object to be processed fails, a fourth response result indicating that the check failed is obtained.

[0162] Specifically, after receiving the write POST request, the storage gateway first performs a legitimacy check on the request interface of the write POST request, and the storage supports such a batch POST interface (the storage that does not support it will consider this to be an illegal request), that is, when the legitimacy check on the request interface of the write POST request fails, a first response result of the legitimacy check on the request interface of the write POST request fails is obtained; then, when the legitimacy check on the request interface of the write POST request passes, a second response result of the legitimacy check on the request interface of the write POST request passes is obtained, and the permissions of the target bucket information and the object to be processed are verified, such as extracting the target bucket information to verify the existence and correctness of the bucket and other permissions, otherwise a request error is directly returned. Finally, the batch storage service of the file is performed, that is, when the permission check on the target bucket information and the object to be processed passes, a third response result of the verification is obtained, and the target bucket information, the object to be processed and the shard data are stored. Specifically, after extracting different file names and corresponding metadata and data, the files are written to the underlying storage medium according to the general storage practices within the storage. For example, small files can be merged into large files and written to ensure the performance of massive files. Regardless of the storage method, when each object to be processed is processed, there will be a corresponding result, which will be reflected in the return error code of each object execution.

[0163] It is understandable that when the permission check of the target bucket information and the object to be processed fails, a fourth response result of failed check is obtained, that is, regardless of whether the check passes or not, the corresponding response result will be returned so that the client can obtain the response result.

[0164] By responding to each pending object separately, it is ensured that the overall request will not fail due to a small number of file write failures. Overall failure means that the client needs to rewrite everything, which is not friendly to the client and is not conducive to performance experience. After the storage gateway completes processing each pending file in the write POST request, the storage gateway saves all response results and encapsulates the response.

[0165] 103: After receiving the response result, if there are pending objects whose request is passed in the response result, then parse the processing result of each pending object whose request is passed in the response result.

[0166] After receiving the response result returned by the storage gateway, the client will check the response result to determine whether the request is successful as a whole, partially successful, or completely failed. When there are pending objects in the response result that have passed the request, the processing result of the request will be parsed.

[0167] In some embodiments, the method further comprises:

[0168] If there is an object to be processed whose request has not been passed in the response result, receiving an error code of the object to be processed whose request has not been passed sent by the storage gateway;

[0169] The failure reason is determined according to the error code, and the failed object to be processed is corrected according to the failure reason, so as to re-create a write POST request according to the corrected object to be processed.

[0170] Specifically, when some checks fail, the response result will contain pending objects whose requests failed. Regardless of whether the checks pass or not, the corresponding response result will be returned, and an error code will be generated if the request fails. The client can determine the reason for the failure based on the error code, and thus correct the pending objects that failed based on the reason for the failure. For example, the request is illegal, or the authority is insufficient, such as exceeding the single limit on the number of objects. In this case, the client can reduce the number of objects so that the verification can pass. Furthermore, the client can rewrite the encapsulated POST request to avoid rewriting the successful object and overwriting the original object. It can also be encapsulated together with other objects to be written to initiate a write POST request. For the storage gateway, it is a stateless write and will not affect the files that have been written successfully.

[0171] The object storage batch processing method, client, and storage gateway of the embodiment of the present invention create a write POST request when the client writes batch objects to the storage gateway, and the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body, and shard data of the objects to be processed, so that one target bucket information corresponds to at least two objects to be processed and their shard data, that is, one write POST request includes multiple objects to be processed and their shard data; the write POST request is sent to the storage gateway, and the storage gateway parses the write request header to request and respond to the target bucket information, the objects to be processed, and the shard data to obtain a response result, writes the target bucket information, the objects to be processed, and the shard data according to the response result, and returns the response result to the client, writes the objects to be processed and the shard data according to the response result, thereby realizing full or partial writing of the objects to be processed and the shard data.

[0172] See also Figure 4 After writing the object to be processed and the shard data, the client can read the object and the shard data stored in the storage gateway. Therefore, the embodiment of the present invention also provides an object storage batch reading method, which is applied to the client, and the method also includes:

[0173] 104: When receiving a read operation instruction for at least one designated object corresponding to the target bucket information, encapsulate a read POST request; the read POST request includes a read request header, target bucket information located in the read request header or the read request line, and a shard range to be read located in the read request body;

[0174] 105: Send the read POST request to the storage gateway, so that the storage gateway responds according to the read POST request after receiving the read POST request; wherein the designated object is the object to be processed that has passed the request in the processing result;

[0175] 106: Read multiple shard data of the specified object according to the response of the storage gateway.

[0176] Specifically, when the client receives a read operation instruction for at least one designated object corresponding to the target bucket information, it encapsulates a read POST request, and then sends the read POST request to the storage gateway. After receiving the read POST request, the storage gateway responds according to the read POST request; wherein the designated object is the object to be processed that has passed the request in the processing result; finally, multiple shard data of the designated object are read according to the response of the storage gateway.

[0177] When the client needs to read an object stored in the storage gateway, it can be a request to read multiple shards of a single object in a single bucket, or it can be a request to read multiple shards of multiple objects in a single bucket; the object to be read is defined as a specified object, and the specified object is the corresponding object to be processed in the processing result. The information of multiple shards is encapsulated in the read POST request, and at least one read POST request corresponding to the specified object is sent to the storage gateway.

[0178] When requesting to read multiple shard data of a single object in a single bucket, the range of shard data offset is obtained. For example, the size of an object is 100 bytes, and the current read operation instruction corresponds to shard data in the range of 0-5 and 10-25. Then, the read POST request will encapsulate the target bucket information, the name of the specified object, and the range of shard data to be read 0-5 and 10-25. That is, the read POST request will not contain actual data, and the content type of the request is fixed. In order to simplify the generation of requests and improve the efficiency of request generation, the request message body in XML format can be used to indicate the file name of the specified object and the range of shards to be read. The maximum number of shards can reach 1000.

[0179] In some of these embodiments, the client encapsulates reading of a POST request, which may include:

[0180] In the read request line, determine the request method as read;

[0181] Put the target bucket information into the read request header;

[0182] A multi-level tag is added to the body of the read request body to identify the specified object and the fragmented data of the specified range of the specified object, wherein the number of the specified object is 1.

[0183] Specifically, Figure 5 As shown, the process of client encapsulation reading POST request is as follows:

[0184] 1. Fill in get in the URL of the request line to indicate read;

[0185] 2. Put the target bucket (bucketname) in the request header;

[0186] 3. Add a blank line;

[0187] 4. The request body is in XML format;

[0188] 5. First-level label <get>;

[0189] 6. Secondary Tags <object>;

[0190] 7. Level 3 Label <key>, intermediate encapsulation object name (Key1);

[0191] 8. Level 3 Labels <versionid>, intermediate package version number (VersionId);

[0192] 9. Level 3 Label <range>, encapsulates the range of multiple fragments (bytes=), separated by commas, such as 0-9, 100-200, 500-600.

[0193] After encapsulating the read POST request, the read POST request is sent to the storage gateway. After receiving the read POST request, the storage gateway responds to the read POST request.

[0194] Correspondingly, when encapsulating the read POST request, the read POST request is encapsulated according to the API interface specification defined by the storage, wherein the API interface is based on the S3 REST API protocol standard.

[0195] In some implementations, after receiving the read POST request, the storage gateway may respond according to the read POST request, which may include:

[0196] Performing a validity check on the read POST request;

[0197] When the read POST request passes the legality check, parsing the body of the read request body to obtain the specified object and the specified range of the specified object;

[0198] Searching for a location corresponding to the specified range in the stored shard data;

[0199] Integrate and / or sort the storage order of the stored shard data according to the positions to obtain a reading order;

[0200] Read the shard data in the specified range according to the reading order;

[0201] The fragment data in the specified range is encapsulated in a response result, and the response result is sent to the client.

[0202] Specifically, the verification of legality may be to first perform a unified request parameter verification process to ensure that the request is legal; and then parse the specific file request in the message body. When obtaining multiple shard data of a single specified object, in order to improve efficiency and reduce data response redundancy, the location of at least one specified object in the read POST request is searched in the stored shard data, and then the shard data corresponding to the specified object is read. Moreover, when reading, the storage order of the stored shard data can be integrated and / or sorted according to the location to obtain a reading order. For example, the range to be read is shards 0-5 and 4-10, which can be merged into one shard 0-10, so that there is no need to repeatedly read or encapsulate shard data of 4-5. For another example, the range of shards to be read is 20-50 and 10-15. If 10-15 is read first and then 20-50 is read, it will be more friendly to the storage bottom layer of the storage gateway, with lower latency, and it can completely avoid reading the entire range of 10-50, that is, it can be sorted, the reading order can be adjusted, and the original reading order is optimized; after obtaining the optimized reading order, the shard data is read according to the reading order, so that the reading efficiency is higher.

[0203] In some implementations, when the number of specified objects is 1, the storage gateway encapsulates the shard data of the specified range in the response result, which may include:

[0204] Put the response result in the response request line;

[0205] Set the fragment separation line in the response request header;

[0206] The body of the response request begins with a fragment separator line;

[0207] Encapsulate the first fragment data of the specified range of the specified object, the first fragment data including the fragment range and data;

[0208] Encapsulate the second fragment data in the specified range until all fragment data in the specified range are encapsulated, and adjacent fragment data are separated by fragment dividing lines;

[0209] Add a slice separator line to end the encapsulation of the slice data in the specified range.

[0210] Specifically, the storage gateway also needs to respond to the client when reading shard data. When the number of specified objects is 1, such as Figure 5 As shown, the storage gateway responds to the client in the following process:

[0211] 1. Put the request response result in the request line;

[0212] 2. Set the fragment boundary in the request header;

[0213] 3. Add a blank line;

[0214] 4. The body of the response begins with a "--dividing line";

[0215] 5. Encapsulate the first shard data: range (Range), data (Content);

[0216] 6. Adding "--dividing line" indicates the existence of the next shard data;

[0217] 7. Encapsulate the second fragment data: range (Range), data (Content);

[0218] 8. Encapsulate other slices in sequence;

[0219] 9. Add "--dividing line--" to indicate the end, and there is no other shard data.

[0220] Finally, after receiving the response, the client parses the shard data in order. If there are already merged shards, it is necessary to determine which shards the requested data is included in according to the shard range, and extract the data to the upper business layer according to its own business needs. The reading of shard data may also fail. You can retry the request according to the error code or contact the operation and maintenance to locate the problem.

[0221] When reading multi-shard data of multiple objects in a single bucket, the multiple different shard data of multiple objects are included in a read POST request and sent to the storage gateway together. The storage gateway responds to the request after processing the entire read POST request, thereby achieving the purpose of batch reading of multiple shard data of multiple objects.

[0222] In some implementations, when the number of the specified objects is greater than 1, that is, when the client wants to read multiple specified objects, the client adds a multi-level tag in the body of the request to identify the specified objects and the fragmented data of the specified range of the specified objects, including:

[0223] Multi-level tags are added to the body of the read request body, and different identifiers are added to different designated objects to identify the designated objects and the fragmented data of the designated range of the designated objects.

[0224] See also Figure 6 In the same bucket, the reading of multi-shard data of multiple objects is an extension of the reading of multi-shard data of a single object. Therefore, the client needs to confirm whether the requested POST file belongs to the same bucket, otherwise the request will fail.

[0225] Specifically, compared with a single specified object, different file identifiers are added to different specified objects. New xml secondary tags can be used to distinguish different specified objects. Then, multiple fragment data of each specified object are included in their respective third-level tags. <range>In addition, the storage interface can support the acquisition of up to 200 specified objects and 200 shard data of each specified object, and after encapsulating the read POST request, it is sent to the storage gateway.

[0226] like Figure 6 As shown, the client constructs a POST request process for reading multiple shard data of multiple specified objects in a bucket as follows:

[0227] 1. Fill in get in the URL of the request line to indicate read;

[0228] 2. Put the target bucket (bucketname) in the request header;

[0229] 3. Add a blank line;

[0230] 4. The request body is in XML format;

[0231] 5. First-level label <get>;

[0232] 6. Secondary Tags <object>;

[0233] 7. Level 3 Label <key>, intermediate encapsulation object name (Key1);

[0234] 8. Level 3 Labels <versionid>, intermediate package version number (VersionId);

[0235] 9. Level 3 Label <range>, encapsulates the range of multiple fragments (bytes=), separated by commas, such as 0-9, 100-200, 500-600;

[0236] 10. Encapsulate other objects Key2 in sequence...

[0237] Then, after receiving the read POST request, the storage gateway needs to parse different specified objects, and then parse multiple shard data of each specified object. Similarly, if there are duplicate files or overlapping shards, the storage will merge them to increase the reading speed and reduce the response packet size. For example, if file a reads 0-5 and 20-25, and file a reads 2-10 and 40-50, they will be merged into file a reading 0-10, 20-25, and 40-50. The reading of each file is separate. The storage first searches for the storage location of the corresponding file data, and then reads the data of different files concurrently. After completion, they are encapsulated separately. Because multiple objects are included, two types of separators need to be implemented in the request body: object separator and shard separator. The object separator is defined in the request header, and the object separator is used in the request body to identify the interval of each object. Each object interval also contains multiple shards, so the shard separator is defined in each object interval. The shard separator of each object is different, and each separate shard is distinguished by the separate separator of the object to which it belongs. After encapsulation, the response is returned to the client.

[0238] When the number of the specified objects is greater than 1, the storage gateway encapsulates the shard data of the specified range in the response result, which may include:

[0239] Put the response result in the response request line;

[0240] Set the fragment separation line in the response request header;

[0241] The body of the response request begins with a fragment separator line;

[0242] Encapsulate the shard data of the specified range of the first specified object, and the shard data of the specified range of the first specified object are separated by shard dividing lines;

[0243] Encapsulate the fragmented data of the specified range of the second specified object until encapsulating the fragmented data of the specified range of all the specified objects; and the fragmented data of the specified range of the second specified object are separated by the fragmentation dividing line;

[0244] Add an object separator line to end the encapsulation of the slice data of the specified range of all specified objects.

[0245] Specifically, see Figure 6 , the corresponding process of storing the response client is as follows:

[0246] 1. Put the request response result in the request line;

[0247] 2. Set the object dividing line (boundary1) in the request header;

[0248] 3. Add a blank line;

[0249] 4. The body of the response begins with "--object dividing line";

[0250] 5. Encapsulate the first object data: object name (Key1), error code (Result), and shard dividing line (boundary2);

[0251] 6. The data starts with "--sharding line";

[0252] 7. Encapsulate the first fragment data: range (Range), data (Content);

[0253] 8. Adding "--dividing line" indicates the existence of the next shard data;

[0254] 9. Encapsulate the second fragment data: range (Range), data (Content);

[0255] 10. Encapsulate other shards in sequence;

[0256] 11. Add "--shard dividing line--" to indicate the end. This object has no other shard data.

[0257] 12. Encapsulate other objects in turn;

[0258] 13. Add "--Object dividing line--" to indicate the end, there are no other objects.

[0259] Finally, after the client receives the response, if it is a successful response, it will parse the object names in the message body in turn, and parse the data of different shards under each object name. Since files and shards may be returned after being merged, they cannot correspond one to one, so the required data must be obtained based on the file name and shard data range. For failed files or shards, the error code can also be identified and retry can be performed.

[0260] The object storage batch processing method of the embodiment of the present invention can write at least one object when writing. Therefore, when the client reads the shard data, similarly, it can read a specified object or multiple specified objects from the storage gateway, send a read POST request to the storage gateway, so that the storage gateway responds according to the read POST request, and reads multiple shard data of the specified object according to the response of the storage gateway, thereby realizing batch reading of multiple shard data. This application can effectively reduce the number of interactions, improve access efficiency, and reduce client resource consumption when it is necessary to write massive objects and read shard data of massive objects.

[0261] See also Figure 7 , Figure 7 The flowchart of the object storage batch writing method provided by the embodiment of the present invention is as follows. The method can be executed by the second controller 23 in the storage gateway. Figure 7 As shown, the method is applied to a storage gateway, and the method includes:

[0262] 201: Receive a write POST request sent by a client, wherein the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body, and shard data of the objects to be processed.

[0263] 202: Parse the write request header to request a response to the target bucket information, the object to be processed, and the shard data, obtain a response result, write the target bucket information, the object to be processed, and the shard data according to the response result, and return the response result to the client.

[0264] The object storage batch writing method, client and storage gateway of the embodiment of the present invention create a write POST request when the client writes batch objects to the storage gateway, and the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body and shard data of the objects to be processed, so that one target bucket information corresponds to at least two objects to be processed and their shard data, that is, one write POST request includes multiple objects to be processed and their shard data; the write POST request is sent to the storage gateway, and the storage gateway parses the write request header to request and respond to the target bucket information, the objects to be processed and the shard data to obtain a response result, writes the target bucket information, the objects to be processed and the shard data according to the response result, and returns the response result to the client, writes the objects to be processed and the shard data according to the response result, thereby realizing full or partial writing of the objects to be processed and the shard data.

[0265] In some embodiments, parsing the write request header to request a response to the target bucket information, the object to be processed, and the shard data, obtaining a response result, writing the target bucket information, the object to be processed, and the shard data according to the response result, and returning the response result to the client includes:

[0266] Parse the write request header, perform a legality check on the write POST request, and obtain a response result corresponding to the object to be processed;

[0267] Placing the response result in a response request line;

[0268] Setting an object dividing line, and placing the object dividing line in the response request header;

[0269] The body of the response request begins with the response dividing line;

[0270] Encapsulate the response result of the first object, the response result including the object name, response code and additional attributes;

[0271] Encapsulate the response result of the second object, and separate the response result of the first object from the response result of the second object by a response dividing line, until the encapsulation of the response results of all objects is completed;

[0272] Add an end dividing line to complete the encapsulation of the response result.

[0273] In some embodiments, the step of performing a validity check on the write POST request to obtain a response result corresponding to the object to be processed includes:

[0274] Verify the legitimacy of the request interface for writing the POST request, wherein, when the legitimacy verification of the request interface for writing the POST request fails, obtain a first response result that the legitimacy verification of the request interface for writing the POST request fails;

[0275] When the legitimacy check of the request interface for writing the POST request passes, a second response result indicating that the legitimacy check of the request interface for writing the POST request passes is obtained, and the target bucket information and the permissions of the object to be processed are verified;

[0276] When the target bucket information and the permission check of the specified object are passed, a third response result indicating that the check is passed is obtained, and the target bucket information, the object to be processed, and the shard data are stored;

[0277] When the permission check of the target bucket information and the object to be processed fails, a fourth response result indicating that the check failed is obtained.

[0278] The present invention also provides an object storage batch reading method, when a client reads data from a storage gateway, the method is applied to the storage gateway, and the method may include:

[0279] 200: After receiving the read POST request sent by the client, respond according to the read POST request, wherein the read POST request includes a read request header, target bucket information located in the read request header or request line, at least one specified object located in the read request body and shard data of the specified object, and the specified object is an object to be processed that passes the request.

[0280] In some embodiments, such as Figure 8 As shown, responding according to the read POST request includes:

[0281] 203: Performing a validity check on the read POST request;

[0282] 204: When the read POST request passes the legality check, parse the body of the read request body to obtain the specified object and the specified range of the specified object;

[0283] 205: Searching for a position corresponding to the specified range in the stored shard data;

[0284] 206: Integrate and / or sort the storage order of the stored shard data according to the positions to obtain a reading order;

[0285] 207: Read the fragment data in the specified range according to the reading order;

[0286] 208: Encapsulate the fragmented data in the specified range into a response result, and send the response result to the client.

[0287] In some implementation modes, the number of the specified objects is 1, and encapsulating the shard data of the specified range in the response result includes:

[0288] Put the response result in the response request line;

[0289] Set the fragment separation line in the response request header;

[0290] The body of the response request begins with a fragment separator line;

[0291] Encapsulate the first fragment data of the specified range of the specified object, the first fragment data including the fragment range and data;

[0292] Encapsulate the second fragment data in the specified range until all fragment data in the specified range are encapsulated, and adjacent fragment data are separated by fragment dividing lines;

[0293] Add a slice separator line to end the encapsulation of the slice data in the specified range.

[0294] In some implementations, if the number of the specified objects is greater than 1, the shard data of the specified range is encapsulated in the response result, including:

[0295] Put the response result in the response request line;

[0296] Set the fragment separation line in the response request header;

[0297] The body of the response request begins with a fragment separator line;

[0298] Encapsulate the shard data of the specified range of the first specified object, and the shard data of the specified range of the first specified object are separated by shard dividing lines;

[0299] Encapsulate the fragmented data of the specified range of the second specified object until encapsulating the fragmented data of the specified range of all the specified objects; and the fragmented data of the specified range of the second specified object are separated by the fragmentation dividing line;

[0300] Add an object separator line to end the encapsulation of the slice data of the specified range of all specified objects.

[0301] Correspondingly, such as Fig. 9 As shown, the embodiment of the present invention further provides a first object storage batch writing device, which can be used for a client. The first object storage batch reading and writing device 700 includes:

[0302] A write request creation module 701 is used to create a write POST request, wherein the write POST request includes a write request header, target bucket information located in the request header or the request line, at least two objects to be processed located in the request body, and shard data of the objects to be processed;

[0303] The sending module 702 is used to send the write POST request to the storage gateway, so that the storage gateway parses the write request header after receiving the write POST request, so as to request a response to the target bucket information, the object to be processed and the shard data, obtain a response result, write the target bucket information, the object to be processed and the shard data according to the response result, and return the response result to the client;

[0304] The response receiving module 703 is used for parsing the processing result of each object to be processed whose request is passed in the response result after receiving the response result.

[0305] In an embodiment of the present invention, when a client writes a batch of objects to a storage gateway, a write POST request is created, and the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body, and shard data of the objects to be processed, so that one target bucket information corresponds to at least two objects to be processed and their shard data, that is, one write POST request includes multiple objects to be processed and their shard data; the write POST request is sent to the storage gateway, and the storage gateway parses the write request header to request a response to the target bucket information, the objects to be processed, and the shard data to obtain a response result, writes the target bucket information, the objects to be processed, and the shard data according to the response result, and returns the response result to the client, writes the objects to be processed and the shard data according to the response result, thereby realizing full or partial writing of the objects to be processed and the shard data.

[0306] In other embodiments, the write request creation module 701 is further used to:

[0307] In the write request line, determine the request method as write;

[0308] Put the target bucket information into a write request header or a write request line;

[0309] Setting an object dividing line in the write request header;

[0310] In the body of the request, the object separator line is used to start the request.

[0311] Encapsulate the first object to be processed, wherein the object to be processed includes the object name, the fragmentation data of the object to be processed, and additional attributes;

[0312] Continue to encapsulate the second object to be processed, and separate the first object to be processed from the second object to be processed by the object dividing line until all objects to be processed are encapsulated;

[0313] Add an end dividing line to complete the encapsulation of the write POST request.

[0314] In other embodiments, the write request creation module 701 is further used to:

[0315] The write POST request is encapsulated according to the API interface specification defined by the storage, wherein the API interface is based on the S3 REST API protocol standard.

[0316] In other embodiments, the first object storage batch reading and writing device 700 further includes a correction module 704, which is used to:

[0317] If there is an object to be processed whose request has not been passed in the response result, receiving an error code of the object to be processed whose request has not been passed sent by the storage gateway;

[0318] The failure reason is determined according to the error code, and the failed object to be processed is corrected according to the failure reason, so as to re-create a write POST request according to the corrected object to be processed.

[0319] like Fig.10 The embodiment of the present invention further provides a first object storage batch reading device, which can be used for a client. The first object storage batch reading device 600 includes:

[0320] The encapsulation read request module 601 is used to encapsulate a read POST request when receiving a read operation instruction for at least one specified object corresponding to the target bucket information;

[0321] The read request sending module 602 is used to send the read POST request to the storage gateway, so that the storage gateway responds according to the read POST request after receiving the read POST request; wherein the specified object is the object to be processed that has passed the request in the processing result;

[0322] The reading module 603 is used to read the multiple shard data of the specified object according to the response of the storage gateway.

[0323] In other embodiments, the encapsulation read request module 601 is further used to:

[0324] In the read request line, determine the request method as read;

[0325] Put the target bucket information into the read request header;

[0326] A multi-level tag is added to the body of the read request body to identify the specified object and the fragmented data of the specified range of the specified object, wherein the number of the specified object is 1.

[0327] In other embodiments, if the number of the specified objects is greater than 1, the encapsulation read request module 601 is further used to:

[0328] Multi-level tags are added to the body of the read request body, and different identifiers are added to different designated objects to identify the designated objects and the fragmented data of the designated range of the designated objects.

[0329] In other embodiments, in the first object storage batch reading device 600, if the read POST request contains at least two specified objects, the number of the specified objects is at most 200, and the number of shard data of each specified object is at most 200; if the read POST request contains only one specified object, the number of shard data of the specified object is at most 1000.

[0330] Correspondingly, such as Fig.11 As shown, the embodiment of the present invention further provides a second object storage batch writing device, which can be used for a storage gateway. The second object storage batch reading and writing device 800 includes:

[0331] The write request receiving module 801 is used to receive a write POST request sent by a client; the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body, and shard data of the objects to be processed;

[0332] The write request response module 802 is used to parse the write request header to request a response to the target bucket information, the object to be processed and the shard data, obtain a response result, write the target bucket information, the object to be processed and the shard data according to the response result, and return the response result to the client.

[0333] In other embodiments, the write request response module 802 is further configured to:

[0334] Parse the write request header, perform a legality check on the write POST request, and obtain a response result corresponding to the object to be processed;

[0335] Placing the response result in a response request line;

[0336] Setting an object dividing line, and placing the object dividing line in the response request header;

[0337] The body of the response request begins with the response dividing line;

[0338] Encapsulate the response result of the first object, the response result including the object name, response code and additional attributes;

[0339] Encapsulate the response result of the second object, and separate the response result of the first object from the response result of the second object by a response dividing line, until the encapsulation of the response results of all objects is completed;

[0340] Add an end dividing line to complete the encapsulation of the response result.

[0341] In other embodiments, the write request response module 802 is further configured to:

[0342] Verify the legitimacy of the request interface for writing the POST request, wherein, when the legitimacy verification of the request interface for writing the POST request fails, obtain a first response result that the legitimacy verification of the request interface for writing the POST request fails;

[0343] When the legitimacy check of the request interface for writing the POST request passes, a second response result indicating that the legitimacy check of the request interface for writing the POST request passes is obtained, and the target bucket information and the permissions of the object to be processed are verified;

[0344] When the target bucket information and the permission check of the specified object are passed, a third response result indicating that the check is passed is obtained, and the target bucket information, the object to be processed, and the shard data are stored;

[0345] When the permission check of the target bucket information and the object to be processed fails, a fourth response result indicating that the check failed is obtained.

[0346] Correspondingly, such as Fig.12 As shown, the embodiment of the present invention further provides a second object storage batch reading device, which can be used for a storage gateway. The second object storage batch reading and writing device 900 includes:

[0347] The read request response module 901 is used to:

[0348] After receiving the read POST request sent by the client, respond according to the read POST request, wherein the read POST request includes a read request header, target bucket information located in the read request header or request line, at least one specified object located in the read request body and shard data of the specified object, and the specified object is the object to be processed that passes the request.

[0349] The read request response module 901 is also used for:

[0350] Performing a validity check on the read POST request;

[0351] When the read POST request passes the legality check, parsing the body of the read request body to obtain the specified object and the specified range of the specified object;

[0352] Searching for a location corresponding to the specified range in the stored shard data;

[0353] Integrate and / or sort the storage order of the stored shard data according to the positions to obtain a reading order;

[0354] Read the shard data in the specified range according to the reading order;

[0355] The fragment data in the specified range is encapsulated in a response result, and the response result is sent to the client.

[0356] In other embodiments, the number of the specified objects is 1, and the read request response module 901 is further used to:

[0357] Put the response result in the response request line;

[0358] Set the fragment separation line in the response request header;

[0359] The body of the response request begins with a fragment separator line;

[0360] Encapsulate the first fragment data of the specified range of the specified object, the first fragment data including the fragment range and data;

[0361] Encapsulate the second fragment data in the specified range until all fragment data in the specified range are encapsulated, and adjacent fragment data are separated by fragment dividing lines;

[0362] Add a slice separator line to end the encapsulation of the slice data in the specified range.

[0363] In other embodiments, the number of the specified objects is greater than 1, and the read request response module 901 is further used to:

[0364] Put the response result in the response request line;

[0365] Set the fragment separation line in the response request header;

[0366] The body of the response request begins with a fragment separator line;

[0367] Encapsulate the shard data of the specified range of the first specified object, and the shard data of the specified range of the first specified object are separated by shard dividing lines;

[0368] Encapsulate the fragmented data of the specified range of the second specified object until encapsulating the fragmented data of the specified range of all the specified objects; and the fragmented data of the specified range of the second specified object are separated by the fragmentation dividing line;

[0369] Add an object separator line to end the encapsulation of the slice data of the specified range of all specified objects.

[0370] It should be noted that the above device can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in the device embodiment, please refer to the method provided in the embodiment of the present application.

[0371] Fig.13 FIG. 1 is a schematic diagram of the hardware structure of the first controller in an embodiment of the client. Fig.13 As shown, the first controller 13 includes:

[0372] One or more first processors 131 and a first memory 132 . Fig.13 In the figure, a first processor 131 and a first memory 132 are taken as an example.

[0373] The first processor 131 and the first memory 132 may be connected via a bus or other means. Fig.13 The example of connecting through bus is taken in the following.

[0374] The first memory 132 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the object storage batch reading / writing method in the embodiment of the present application (for example, the attached Fig. 9 The write request creation module 701, the sending module 702, the response receiving module 703, the correction module 704 and the attached Fig.10 The first processor 131 executes various functional applications and data processing of the controller by running the non-volatile software programs, instructions and modules stored in the first memory 132, that is, the object storage batch reading / writing method of the above method embodiment is implemented.

[0375] The first memory 132 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required by at least one function; the data storage area may store data created according to the use of the first object storage batch read-write device, etc. In addition, the first memory 132 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the first memory 132 may optionally include a memory remotely arranged relative to the first processor 131, and these remote memories may be connected to the client via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0376] The one or more modules are stored in the first memory 132, and when executed by the one or more first processors 131, the object storage batch processing method in any of the above method embodiments is executed, for example, the object storage batch processing method described above is executed. Figure 1 Steps 101 to 103 of the method, Figure 4 Method steps 104-106 in; implementing Fig. 9 Modules 701-704 in Fig.10 The functions of modules 601-603 in FIG.

[0377] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0378] Fig.14 FIG. 1 is a schematic diagram of the hardware structure of the second controller in an embodiment of the client. Fig.14 As shown, the second controller 23 includes:

[0379] One or more second processors 231 and a second memory 232 . Fig.14 In the figure, a second processor 231 and a second memory 232 are taken as an example.

[0380] The second processor 231 and the second memory 232 may be connected via a bus or other means. Fig.14 The example of connecting through bus is taken in the following.

[0381] The second memory 232 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as program instructions / modules corresponding to the object storage batch writing / reading method in the embodiment of the present application (for example, the attached Fig.11 The write request receiving module 801, the write request responding module 802 and Fig.12 The second processor 231 executes various functional applications and data processing of the controller by running the non-volatile software programs, instructions and modules stored in the second memory 232, that is, the object storage batch writing / reading method of the above method embodiment is implemented.

[0382] The second memory 232 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required by at least one function; the data storage area may store data created according to the use of the second object storage batch write / read device, etc. In addition, the second memory 232 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the second memory 232 may optionally include a memory remotely arranged relative to the second processor 231, and these remote memories may be connected to the client via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0383] The one or more modules are stored in the second memory 232, and when executed by the one or more second processors 231, the object storage batch processing method in any of the above method embodiments is executed, for example, the object storage batch processing method described above is executed. Figure 7-8 Steps 201 to 202, and steps 203 to 208 of the method; implementing Fig.11 Modules 801-802 and Fig.12 The function of module 901 in.

[0384] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0385] The embodiment of the present application provides a non-volatile computer-readable and writable storage medium, wherein the computer-readable and writable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example Fig.13 and 14 A first processor 131 / a second processor 231 in the embodiment may enable the one or more first processors 131 / a second processor 231 to execute the object storage batch reading / writing method in any of the above method embodiments, for example, to execute the above described Figure 1 Steps 101 to 103 of the method, Figure 4 Steps 104-106 of the method, Figure 7 Method steps 201 to 202; implement Fig. 9 Modules 701-704 in Fig.10 Modules 601-603 in Fig.11 Modules 801-802 and Fig.12 The function of module 901 in.

[0386] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0387] Through the description of the above embodiments, a person of ordinary skill in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, or of course by hardware. A person of ordinary skill in the art can understand that all or part of the processes in the above embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0388] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they may still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.< / range> < / versionid> < / key> < / object> < / get> < / range> < / range> < / versionid> < / key> < / object> < / get>

Claims

1. A method for batch reading of object storage, characterized in that: Applied to a storage gateway, the method includes: After receiving a read POST request sent by a client, respond according to the read POST request, wherein the read POST request includes a read request header, target bucket information located in the read request header or request line, a specified object located in the read request body, and shard data of the specified object, and the specified object is an object to be processed through the request, and the number of the specified objects is greater than 1; Wherein, responding according to the read POST request includes: Performing a validity check on the read POST request; When the read POST request passes the legality check, parsing the body of the read request body to obtain the specified object and the specified range of the specified object; Searching for a location corresponding to the specified range in the stored shard data; Integrate and / or sort the storage order of the stored shard data according to the positions to obtain a reading order; Read the multiple shard data in the specified range according to the reading order; Put the response result in the response request line; Set the fragment separation line in the response request header; The body of the response request begins with a fragment separator line; Encapsulate the shard data of the specified range of the first specified object, and the shard data of the specified range of the first specified object are separated by shard dividing lines; Encapsulate the fragmented data of the specified range of the second specified object until encapsulating the fragmented data of the specified range of all the specified objects; and the fragmented data of the specified range of the second specified object are separated by the fragmentation dividing line; An object dividing line is added to end the encapsulation of the fragment data of the specified range of all specified objects, and the response result is sent to the client.

2. A method for batch processing and writing of object storage, characterized in that: Applied to a storage gateway, the method includes: Receive a write POST request sent by a client; the write POST request includes a write request header, target bucket information located in the request header or request line, at least two objects to be processed located in the request body, and shard data of the objects to be processed; Parse the write request header, and verify the legitimacy of the request interface of the write POST request, wherein, when the legitimacy check of the request interface of the write POST request fails, obtain a first response result that the legitimacy check of the request interface of the write POST request fails; When the legitimacy check of the request interface for writing the POST request passes, a second response result indicating that the legitimacy check of the request interface for writing the POST request passes is obtained, and the target bucket information and the permissions of the object to be processed are verified; When the target bucket information and the permission check of the designated object are passed, a third response result indicating that the check is passed is obtained, and the target bucket information, the object to be processed, and the shard data are stored; When the target bucket information and the permission check of the object to be processed fail, obtaining a fourth response result indicating that the check failed; Placing the response result in a response request line; Setting an object dividing line, and placing the object dividing line in a response request header; The body of the response request begins with the response dividing line; Encapsulate the response result of the first object, the response result including the object name, response code and additional attributes; Encapsulate the response result of the second object, and separate the response result of the first object from the response result of the second object by a response dividing line, until the encapsulation of the response results of all objects is completed; Add an end dividing line to complete the encapsulation of the response result.

3. A storage gateway, characterized in that: The storage gateway comprises: at least one second processor, and A second memory, wherein the second memory is communicatively connected to the second processor, and the second memory stores instructions executable by the at least one second processor, wherein the instructions are executed by the at least one second processor so that the at least one processor can execute the method according to claim 1 or 2.

4. A non-volatile computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by the terminal, the storage gateway executes the method according to claim 1 or 2.

Images