Object storage management method and device, gateway equipment and storage system

Abstract

The invention relates to an object storage management method and device, gateway equipment and a storage system. The storage gateway receives the current data and writes the current data into the corresponding block device; the storage gateway comprises a plurality of block devices, and each block device divides a data object according to a continuous offset range with a fixed length; setting a unique storage name for the current data to form a to-be-transmitted data object; after a preset uploading condition is met, uploading the to-be-uploaded data object in each block device to an object storage system; and after uploading is completed, generating index data carrying version information or time information according to the block device address and the unique storage name of each uploaded data object, and updating the index file. According to the method, the storage gateway realizes uploading processing and data management of the multi-version object, and is not limited by whether different object storage products support a multi-version control function, so that the storage gateway is flexibly adapted to all object storage products, and the compatibility with various object storage products is greatly improved.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to an object storage management method, apparatus, gateway device and storage system. Background Art

[0002] Object storage is a storage method that uses objects as the basic storage unit and manages data through objects. Currently, various object storage products have been designed by manufacturers. Most object storage products have a common feature of supporting multi-version control, that is, for a certain object, multiple versions are allowed to exist at the same time, and operations such as downloading and deleting a specific version of the object can be performed by specifying version information.

[0003] In order to transfer local data to object storage products, manufacturers design matching storage gateways and use their own object storage as the backend storage layer to manage and transmit object data. However, in this method, users cannot replace the backend storage layer products according to their own usage requirements, and the flexibility is poor. If the storage gateway can adapt to all object storage products and allow users to choose object storage products as the backend storage layer according to their own usage requirements, the flexibility and scalability of object storage will be improved. However, some object storage products on the market have multi-version control functions, while some object storage products do not have multi-version control functions, which makes product compatibility difficult. Summary of the invention

[0004] Based on this, it is necessary to provide an object storage management method, apparatus, gateway device and storage system to address the above technical problems, which can realize upload processing and data management of multi-version objects on the storage gateway, so that the storage gateway is compatible with all object storage products.

[0005] In a first aspect, the present application provides an object storage management method, which is applied to a storage gateway, including:

[0006] Receive current data and write the current data to the corresponding block device; the storage gateway includes multiple block devices, each of which divides data objects according to a continuous and fixed-length offset range;

[0007] Set a unique storage name for the current data to form the data object to be transmitted;

[0008] When the preset upload conditions are met, the data objects to be transmitted in each block device are uploaded to the object storage system;

[0009] After the upload is completed, index data is generated based on the block device address and unique storage name of each uploaded data object; the index data carries version information or time information;

[0010] Update the index file based on the index data.

[0011] In one embodiment, the index file includes a position index area and a version index area, the position index area records the position information of each transmitted data object in the block device, and the version index area stores a plurality of index data; the method further includes:

[0012] Receive a data read request;

[0013] According to the data read request, read data from the corresponding target block device;

[0014] In the case where the data is not read locally, searching in the location index area according to the location information of the target block device to determine whether the target data object corresponding to the target block device is a transmitted data object;

[0015] In the case where it is determined that the target data object is a transmitted data object, determining a unique storage name of the target data object from the version index area;

[0016] Obtain data from the object storage system based on the determined unique storage name;

[0017] The acquired data is returned to the sender of the data read request.

[0018] In one embodiment, after uploading the data objects to be transmitted in each block device to the object storage system, the method further includes:

[0019] For each data object to be transmitted, if the current data object to be transmitted is uploaded, the corresponding data object information is recorded in the progress status file.

[0020] In one embodiment, the method further comprises:

[0021] If a failure occurs, the uploading of the data objects to be transmitted in each device will be stopped;

[0022] When it is determined that the fault has been cleared, the progress status file is read to obtain data object information of the transmitted data object;

[0023] According to the acquired data object information, continue to upload the data objects to be transmitted in each block device.

[0024] In one embodiment, after updating the index file according to the index data, the method further includes:

[0025] Generate index object according to index data;

[0026] Upload the index object to the object storage system.

[0027] In one embodiment, the method further comprises:

[0028] Determine the old version index data according to the version information or time information carried by each index data in the index file;

[0029] Delete the data objects corresponding to the old version of index data in the object storage system;

[0030] Delete the old version of the index object stored in the object storage system.

[0031] In one embodiment, the method further comprises:

[0032] In case of local data corruption, scan each index object stored in the object storage system;

[0033] Determine at least one latest index object according to the version information or time information carried by each index object;

[0034] Read the unique storage name of the data object to be restored in at least one latest index object;

[0035] Obtain target data from the object storage system based on the unique storage name read;

[0036] Perform local data recovery based on the acquired target data.

[0037] In a second aspect, the present application further provides an object storage management device, which is applied to a storage gateway, including:

[0038] The receiving module is used to receive the current data and write the current data into the corresponding block device; the storage gateway includes multiple block devices, each of which divides the data object according to a continuous and fixed-length offset range;

[0039] The naming module is used to set a unique storage name for the current data to form the data object to be transmitted;

[0040] The upload module is used to upload the data objects to be transmitted in each block device to the object storage system when the preset upload conditions are met;

[0041] The index generation module is used to generate index data according to the block device address and unique storage name of each uploaded data object after the upload is completed; the index data carries version information or time information; and the index file is updated according to the index data.

[0042] In a third aspect, the present application further provides a gateway device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in the first aspect above are implemented.

[0043] In a fourth aspect, the present application further provides a storage system, the storage system comprising: a storage gateway and an object storage system; wherein:

[0044] The storage gateway is used to execute the steps of the method described in the first aspect above.

[0045] In the above-mentioned object storage management method, apparatus, gateway device and storage system, the storage gateway receives the current data and writes the current data into the corresponding block device; the storage gateway includes multiple block devices, each block device divides the data object according to a continuous and fixed-length offset range; a unique storage name is set for the current data to constitute the data object to be transmitted; when the preset upload conditions are met, the data object to be transmitted in each block device is uploaded to the object storage system; after the upload is completed, index data is generated according to the block device address and unique storage name of each transmitted data object; the index data carries version information or time information; and the index file is updated according to the index data. In the above-mentioned manner, the storage gateway sets a unique storage name for each version of the data object, and based on the version information or time information of the index data, the upload processing and data management of multi-version objects are realized, which is no longer restricted by whether different object storage products themselves support multi-version control functions, so that the storage gateway can flexibly adapt to all object storage products, and the compatibility with various object storage products is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] FIG1 is a diagram of an application environment of an object storage management method according to an embodiment;

[0048] FIG2 is a schematic diagram of a flow chart of an object storage management method in one embodiment;

[0049] FIG3 is a schematic diagram of a flow chart of an object storage management method in another embodiment;

[0050] FIG4 is a structural block diagram of an object storage management device in one embodiment;

[0051] FIG. 5 is a diagram showing the internal structure of a gateway device in one embodiment. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It is understandable that the storage gateway involved in this embodiment, namely the Cloud Storage Gateway, serves between local applications and remote cloud storage, and transmits local data to the cloud. When object storage is used as a cloud service, the objects transmitted to the cloud need to exist in the form of multiple versions for the following two reasons:

[0054] 1. Ensure crash consistency. If the object does not exist in the form of multiple versions, when the new version of an object is uploaded, it will overwrite the old version, which may break the principle of crash consistency. After a batch of data is written, the data will be uploaded. Only when all the data are uploaded successfully, it means that the upload of this batch of data is complete. If only half of this batch of data is uploaded, it means that this batch of data has not been uploaded successfully, and the last batch of successfully uploaded data will be used as the most valid data on the cloud. If the old object is directly overwritten with the new object, when half of the data is uploaded, the principle of data crash consistency will not be met. Therefore, when a new batch of data is not fully uploaded, it is necessary to retain both the old and new versions of the data objects. Only when the new batch of data is fully uploaded can the old version of the object be deleted.

[0055] 2. Solve the disorder problem of concurrent uploads. When uploading an object, if it is blocked in a certain step of network transmission and returns a failure to the upper layer after the timeout period, and then an updated version of the object is uploaded, and the request that was previously blocked in network transmission continues to be transmitted after a period of time, if the object does not exist in multiple versions, the old version will overwrite the new version, resulting in disorder.

[0056] Therefore, when using object storage as a cloud service, objects transmitted to the cloud need to exist in the form of multiple versions to ensure crash consistency and solve the disorder problem of concurrent uploads.

[0057] Currently, some object storage products have multi-version control functions, which mainly use randomly generated and globally unique strings to represent the version ID of the object. It is impossible to directly determine the newness of the object by comparing the version ID. In addition, the string is long, so it will take up more space whether it is stored in memory or in persistent storage. In addition to this limitation, some object storage products do not have multi-version control functions, which makes product compatibility difficult.

[0058] In order to solve the above problems, this application proposes an object storage management method, apparatus, gateway device and storage system. Instead of using the multi-version control function provided by the object storage product itself, a solution for managing multi-version objects on the object storage system is designed on the storage gateway to achieve data management, cloud upload, cloud download, garbage cleaning, breakpoint resume and other functions. By adopting this method, the storage gateway can flexibly adapt to all object storage products, and is no longer restricted by whether different object storage products themselves support multi-version control functions, and the compatibility with various object storage products is greatly improved.

[0059] It should be noted that the embodiments of the present application mainly involve two types of storage objects: data objects and index objects. Data objects refer to the objects corresponding to the data written into the storage gateway by the application layer after it is uploaded to the object storage system. Index objects refer to the objects corresponding to the data object related information uploaded to the object storage system. Both types of objects need to be stored in the object storage system in a multi-version manner. The embodiments of the present application do not use the multi-version control function provided by the object storage itself, but implement a multi-version control solution on the storage gateway based on the object storage management method provided by the present application.

[0060] The object storage management method provided in the embodiment of the present application can be applied in the application environment shown in FIG1 . The storage system includes a storage gateway 10 and an object storage system 20; the user submits a data read and write request through the client, the storage gateway 10 receives the data read and write request, executes the object storage management method provided in the present application, and uploads the data provided by the client to the object storage system 20 or reads the data from the object storage system 20.

[0061] In an exemplary embodiment, in combination with FIG. 1 and as shown in FIG. 2 , an object storage management method is provided, which is described by taking the method applied to the storage gateway 10 in FIG. 1 as an example, including:

[0062] Step 202, receiving current data, and writing the current data into a corresponding block device; the storage gateway includes multiple block devices, and each block device divides data objects according to a continuous and fixed-length offset range.

[0063] Among them, the current data is the data provided by the client that needs to be uploaded to the object storage system. The storage gateway of this embodiment is implemented based on block devices, and the current data written is split according to the continuous and fixed-length offset of the block device. For example, if the written data will be split according to the continuous 1M length of the block device, the result of the split is: [0,1M), [1M,2M), [2M,3M)..., that is, the data within each 1M continuous offset range will be constructed into a data object to be uploaded.

[0064] Optionally, the storage gateway receives current data from the client, parses the current data, determines address information to be written, and writes the current data into an offset range corresponding to the block device according to the address information.

[0065] Step 204, set a unique storage name for the current data to form a data object to be transmitted.

[0066] Among them, this embodiment uses different names, i.e., unique storage names, for different data objects and different versions of the same data object. The traditional method generally requires that the version number be reflected in the object name. Compared with this, this embodiment does not limit the specific naming method of the data object. It only needs to satisfy the need to use different names for different data objects and different versions of the same data object. Users can customize the naming rules according to their own needs.

[0067] The naming process of this embodiment is explained below with examples: the two pieces of information, the starting offset and the version number of the block device, are used for naming: startOffset-version. If the written data is split according to the continuous 1M length of the block device, the data objects after the split are named: 0-version1, 1048576-version2, 2097152-version3... When new data is written within a certain offset range, it will still be split according to the continuous 1M length of the block device and transmitted to the cloud again. If new data is written within the range of [0,1M) and [2M,3M), the data object names specified when transmitting to the cloud are 0-version4 and 2097152-version5 respectively. 0-version1 and 0-version4 are different versions of the same data object, 2097152-version3 and 2097152-version5 are different versions of the same data object, 0-version4 is higher than 0-version1, and 2097152-version5 is higher than 2097152-version3. It should be noted that this implementation does not have any requirements on the numerical order of version4 and version1, version5 and version3. If the version number is represented by a long type number, it is not required that the number corresponding to version4 is greater than the number corresponding to version1. It is only necessary that version4 and version1 are represented by different numbers. Because version is only used to implement different object names for different versions of the same data object, in actual applications, the version number can be incremented for business convenience, but it is not a mandatory requirement.

[0068] It is understandable that this embodiment does not utilize the multi-version control function of the object storage itself, but sets the data object name on the storage gateway so that different data objects and different versions of the same data object have different names. It has the following advantages: 1. The form of the version number can be controlled according to its own needs. For example, it can be named using a long type number or a string with a certain pattern, which is convenient for implementing multi-version control according to business scenarios or its own needs. 2. If the multi-version control function of the current object storage product itself is used, the storage will waste space due to the long string length; through customized naming rules, the length of the unique storage name or the version information carried can be controlled, thereby saving storage space. 3. Since the multi-version control function provided by the object storage product itself is no longer used, it can be better compatible with all object storage products currently on the market, even object storage products that do not support multi-version control can be well compatible. The storage gateway can flexibly adapt to all object storage products as the backend storage layer. 4. It should be noted that this embodiment does not have any requirements on the form of data object version information, which can be in any form such as digital form, string form, etc., and there is no requirement on the size order of version numbers. It does not need to meet the order requirements such as increasing or decreasing. It only needs to meet the requirements that different data objects and different versions of the same data object are named with different names. 5. Furthermore, this embodiment does not limit the need for the unique storage name of the data object to carry version information, as long as different versions of the same data object have different object names. For example, a data object has two versions, and the object names of these two versions of the object on the cloud are "abc" and "123" respectively. These two names have nothing to do with each other, but they can distinguish different versions of the same data object.

[0069] Step 206: When the preset upload conditions are met, the data objects to be transmitted in each block device are uploaded to the object storage system.

[0070] Among them, the storage gateway uploads data to the object storage system in batches. It can be understood that the preset upload condition is a rule set in advance. Optionally, the preset upload condition corresponds to a preset size of each batch of data. If the data volume of the data objects to be transmitted in each block device of the storage gateway reaches the preset size of each batch of data, it is determined that the preset upload condition is met. Optionally, the preset upload condition corresponds to a preset upload time for each batch of data. If the cumulative time of the data objects to be transmitted reaches the preset upload time for each batch of data, it is determined that the preset upload condition is met.

[0071] It is understandable that the current batch of data objects to be transmitted is uploaded to the object storage system. The object storage system can be any current object storage product, with the following functions: 1. Ability to read objects (GetObject); 2. Ability to store objects (PutObject); 3. Ability to view object lists (ListObjects); 4. Ability to delete objects (DeleteObject); 5. Strong consistency, that is, when an object is stored (Put) in the object storage, the object is visible to all subsequent Get and List requests.

[0072] Step 208, after the upload is completed, index data is generated according to the block device address and unique storage name of each uploaded data object; the index data carries version information or time information.

[0073] Among them, after the data objects to be transmitted are successfully uploaded, it is necessary to record the relevant information of the transmitted data objects through indexes, which may include the location information of the transmitted data objects, the version information of the transmitted data objects, etc. In this embodiment, index data is generated according to the block device address and unique storage name of each transmitted data object.

[0074] It is understandable that different versions of index objects also need to be distinguished by different names or tags. Since the naming rule of data objects is: different names can be used to represent different data objects and different versions of the same data object, this embodiment requires index objects to identify the newness and oldness of versions. Therefore, when generating index objects, the index objects are named or attributes are added so that they carry version information or time information. For example, the newness and oldness of the index object version can be indicated by reflecting ordered version information in the request object name; for another example, the naming method is not limited but the version information is recorded in the meta of the index object; for another example, the naming method is not limited but a timestamp is added to the attributes of the index object.

[0075] Among them, the index object is used to manage the data objects on the object storage system, record the block device address and unique storage name of the data object on the cloud, and carry version information or time information, based on which the new and old of the same data object on the cloud can be identified.

[0076] In an optional implementation, the index object also needs to be stored in the cloud in multiple versions, so different versions of the same index object also need to have different names, and the new and old versions can be identified. Multi-version storage of index objects in the cloud is achieved by reflecting the version number information in the name of the index object, and the order of the version number is required, that is, the version number on the name of the new version index object needs to be greater than the version number on the name of the old version index object, so that the new and old index objects can be identified through the version number information on the index object name.

[0077] In an optional implementation, the content recorded by the index includes two parts: a location index area and a version index area. Referring to Table 1, Table 1 shows an index record content. In this embodiment, the unique storage name and version of the data object on the cloud are mainly associated through the version index area. A set of startOffset and name in the version index area represents the block device address and unique storage name of a data object on the cloud. The block device address of this embodiment uses the starting offset (startOffset). According to the foregoing content, this embodiment is based on block device implementation, and the written data will be split into data objects according to continuous and fixed-length offsets. Therefore, the starting offset of the data object is used as a mark to identify the data object, and the data object can be associated with the name on the cloud.

[0078] Table 1:

[0079]

[0080] The same data object may have different versions on the cloud, and you also need to know the versions represented by different names of the same data object on the cloud. Whenever a batch of new data is uploaded, the index will be updated. The version index area in the latest index records the latest valid version of the data object on the cloud.

[0081] Referring to Table 2 and Table 3, Table 2 shows the content recorded in the version index area of ​​the old index; Table 3 shows the content recorded in the version index area of ​​the new index.

[0082] Table 2:

[0083]

[0084] Table 3:

[0085]

[0086] According to the content recorded in the version index area in the index, we can know that the data objects named name4 and name1 represent the new and old versions of the same data object, and the data objects named name5 and name3 represent the new and old versions of the same data object. It can be seen that since the index object stores the block device address and unique storage name of each transmitted data object, and also carries version information or time information, based on the index object, the unique storage name and version of the data object on the cloud can be associated, thereby realizing version management of the data object. That is, a data object is associated with a unique storage name on the cloud through an index, and then the new and old versions of different names of the same data object on the cloud are identified through the new and old index, based on which the indexing and management of data objects on the cloud is realized.

[0087] Optionally, the old version data objects recorded in the old index can be cleaned up at an appropriate time, thereby saving storage space and improving storage resource utilization.

[0088] Step 210: Update the index file according to the index data.

[0089] The index file is a local file of the storage gateway. Through the index file, the storage gateway can manage the information of the data objects of the object storage system based on the index file.

[0090] In the above object storage management method, the storage gateway receives the current data and writes the current data to the corresponding block device; the storage gateway includes multiple block devices, each block device divides the data object according to a continuous and fixed-length offset range; a unique storage name is set for the current data to form a data object to be transmitted; when the preset upload conditions are met, the data object to be transmitted in each block device is uploaded to the object storage system; after the upload is completed, index data is generated according to the block device address and unique storage name of each transmitted data object; the index data carries version information or time information; according to the index data, the index file is updated. In the above manner, the storage gateway sets a unique storage name for each version of the data object, and based on the version information or time information of the index data, the upload processing and data management of multi-version objects are realized, which is no longer restricted by whether different object storage products themselves support multi-version control functions, so that the storage gateway can flexibly adapt to all object storage products, and the compatibility with various object storage products is greatly improved.

[0091] In an exemplary embodiment, as shown in FIG3 , the index file includes a position index area and a version index area, the position index area records the position information of each transmitted data object in the block device, and the version index area stores a plurality of index data; the method further includes:

[0092] Step 302: Receive a data read request.

[0093] Step 304: Read data from the corresponding target block device according to the data read request.

[0094] After receiving the data read request sent by the client, the data is first read from the local device. Optionally, the data read request is analyzed to determine the address information of the requested data, and the data is read from the offset range corresponding to the target block device according to the address information.

[0095] Step 306: If no data is read locally, a search is performed in the location index area according to the location information of the target block device to determine whether the target data object corresponding to the target block device is a transmitted data object.

[0096] Among them, referring to Table 1 above, the location index area records multiple sets of offsets and lengths, which are used to indicate which locations in the block device have uploaded data to the cloud. If the data to be read does not exist locally, it is necessary to go to the cloud to obtain the data. According to the location index area of ​​the local index file, it is determined whether the location to be read has uploaded data.

[0097] Step 308: When it is determined that the target data object is a transmitted data object, a unique storage name of the target data object is determined from the version index area.

[0098] Step 310: Obtain data from the object storage system according to the determined unique storage name.

[0099] If data has been uploaded to the location to be read, query the version index area to obtain the unique storage name of the data object on the cloud, and then download the data object to be read from the cloud according to the unique storage name.

[0100] Step 312: Return the acquired data to the sender of the data read request.

[0101] In this embodiment, the unique storage name of the data object on the cloud can be determined through the location index area and the version index area, so that data can be transferred to the cloud. First, the data is read from the local storage gateway. If the data is not read, it is determined whether the data to be read has been uploaded. If it is determined that the data has been uploaded to the object storage system, the relevant data is read from the object storage system, which can improve the data reading efficiency.

[0102] In an exemplary embodiment, after step 206, the method further includes: for each data object to be transmitted, if the current data object to be transmitted is uploaded, recording corresponding data object information in a progress status file.

[0103] Among them, the progress status file is used to record the progress of the upload. If a failure occurs in the middle of uploading a batch of data, the progress status file is set to continue the original progress for uploading after the failure is restored. Exemplarily, referring to Table 4, Table 4 shows a data structure written in a progress status file. The progress status file adopts an appended file structure. Whenever a data object is uploaded, a set of data structures shown in Table 4 will be recorded in the progress status file, indicating that the data object has been uploaded in this batch of data uploads. startOffset represents the starting offset corresponding to the data object, name represents the name corresponding to this object, and multiple groups of luOffset+length indicate that the data object uploaded this time contains data in the offset range of [luOffset, luOffset+length-1].

[0104] Table 4:

[0105]

[0106] In this embodiment, a progress status file is provided to record the upload progress, so that the breakpoint resume function can be implemented.

[0107] In an exemplary embodiment, the method further includes: if a fault occurs, stopping uploading the data objects to be transmitted in each block device; when it is determined that the fault has been cleared, reading the progress status file to obtain data object information of the transmitted data objects; and continuing to upload the data objects to be transmitted in each block device based on the obtained data object information.

[0108] It is understandable that the present embodiment uses batch data to upload data, and usually the index object is not updated every time a data object is uploaded. In order to improve efficiency, the index object is updated once after a batch of data is uploaded, wherein data accumulated over a period of time or a certain amount of data is called a batch of data. Therefore, if a batch of data has not been uploaded and the index has not been updated, if a fault occurs at this time, the current batch of uploaded data objects have not been recorded in the index object, and the uploaded data objects are invalid, and need to be re-uploaded when the service is restored. In the process of uploading data objects, the progress of the upload is continuously recorded through the progress status file in the present embodiment. After the fault is restored, the previous upload progress can be identified through the progress status file, and the previous upload progress can be continued, thus avoiding the retransmission of certain data objects, thereby improving efficiency. Compared with updating the index object after each data object is uploaded, the advantages of updating the progress status file after each data object is uploaded are: the progress status file is a local file, and local writing operations are much more time-saving than uploading index objects to the cloud; the progress status file is a file structure that can be appended, while the structure of the index object is relatively fixed, and the status of all data objects must be recorded completely. If the index object is updated during the process of uploading batch data, it will greatly affect efficiency.

[0109] In an exemplary embodiment, after step 210, the method further includes: generating an index object according to the index data; and uploading the index object to the object storage system.

[0110] When updating the index, the local index file will be updated first, and then it will be constructed into an index object and uploaded to the cloud. Index objects also need to be stored in the cloud in multiple versions. The updated index will be uploaded to the cloud as an updated version, that is, the latest index object records the latest data objects, and the older index objects contain older data objects. Local index files are mainly used to speed up the response speed of read requests, because when responding to read requests, it is necessary to download the corresponding data objects from the cloud according to the queried index records. The speed of reading index files from the local is better than the speed of downloading index objects from the cloud; the index objects on the cloud are mainly used for disaster recovery from the cloud when local data is damaged.

[0111] In an exemplary embodiment, the method also includes: determining old version index data based on version information or time information carried by each index data in the index file; deleting the data object corresponding to the old version index data in the object storage system; and deleting the old version index object stored in the object storage system.

[0112] Among them, for the index data in the index, the old version data objects and the old version index objects are deleted regularly or based on the pre-set cleaning rules at the appropriate time. Specifically, the new and old versions of the index data are identified according to the version information or time information carried by each index data in the index file. If the data object corresponding to the current index data appears as an updated version in the new version index data, the data object corresponding to the current index data is determined as the old version data object. The data objects corresponding to the old version index data in the object storage system are deleted; the old version index objects stored in the object storage system are deleted. For example: the old version index object records the information of a0, b0, and c0 data objects. At this time, the a1 data object is uploaded. The new version index object will record the information of a1, b0, and c0 data objects. The storage gateway can delete the a0 data object and the old version index object stored in the object storage system. In this way, it is possible to clean up junk data objects on the cloud in a timely manner.

[0113] In an optional implementation, the object storage management method includes the following steps:

[0114] 1. The client writes data to the cloud storage gateway. When the amount of written data accumulates to a certain value, it plans to start uploading;

[0115] 2. Construct the data object to be uploaded and name it;

[0116] 3. Use PutObject of object storage to transfer data objects to the cloud;

[0117] 4. Record the uploaded data objects in the progress status file;

[0118] 5. After all the data is transferred to the cloud, update the local index file;

[0119] 6. Use PutObject of object storage to transfer the index object to the cloud;

[0120] 7. Then, the client continues to write data to the gateway. When the amount of written data accumulates to a certain value, the plan starts to upload;

[0121] 8. Construct the data objects to be uploaded and name them. If the data objects in this batch of uploaded data include the same data objects as those in step 2, different object names are required to ensure that different versions of the same data object exist simultaneously in the cloud;

[0122] 9. Use PutObject of object storage to transfer data objects to the cloud;

[0123] 10. Record the uploaded data objects in the progress status file;

[0124] 11. After all the data is transferred to the cloud, update the local index file;

[0125] 12. Use PutObject of the object storage to transfer the index object to the cloud. If the uploaded index object contains the same index object as in step 6, the index object needs to be named with an increasing version number.

[0126] 13. Based on the old index object, delete the old data object on the cloud through the DeleteObject of the object storage at an appropriate time;

[0127] 14. Delete the old index objects on the cloud.

[0128] In an exemplary embodiment, the method also includes: in the event of local data corruption, scanning each index object stored in the object storage system; determining at least one latest index object based on version information or time information carried by each index object; reading the unique storage name of the data object to be recovered in at least one latest index object; obtaining target data from the object storage system based on the read unique storage name; and performing local data recovery based on the obtained target data.

[0129] When there is a problem with the local data or the data is damaged, the local data can be restored from the cloud. Specifically, all index objects are scanned in the cloud, and the new version of the index object is identified. According to the data objects recorded in the index object, the data is obtained and restored.

[0130] In an optional implementation, when the local data of the storage gateway is damaged, the data on the cloud can be used for recovery. The specific process is as follows:

[0131] 1. Scan the index objects on the cloud through ListObjects of object storage;

[0132] 2. Identify the old and new versions of the index object based on the version information on the index object name;

[0133] 3. Read the relevant information recorded in the new version index object and identify the valid data objects on the cloud;

[0134] 4. Recover the data contained in the cloud data object through GetObject of object storage.

[0135] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0136] Based on the same inventive concept, the embodiment of the present application also provides an object storage management device for implementing the object storage management method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more object storage management device embodiments provided below can refer to the limitations of the object storage management method above, and will not be repeated here.

[0137] In an exemplary embodiment, in combination with FIG. 1 , as shown in FIG. 4 , an object storage management device is provided, which is described by taking the application of the device to the storage gateway 20 in FIG. 1 as an example, including:

[0138] The receiving module 402 is used to receive current data and write the current data into a corresponding block device; the storage gateway includes multiple block devices, and each block device divides data objects according to a continuous and fixed-length offset range.

[0139] The naming module 404 is used to set a unique storage name for the current data to form a data object to be transmitted.

[0140] The upload module 406 is used to upload the data objects to be transmitted in each block device to the object storage system when the preset upload conditions are met.

[0141] The index generation module 408 is used to generate index data according to the block device address and unique storage name of each uploaded data object after the upload is completed; the index data carries version information or time information; and the index file is updated according to the index data.

[0142] In the above-mentioned object storage management device, the storage gateway sets a unique storage name for each version of the data object, and realizes the upload processing and data management of multi-version objects based on the version information or time information of the index data. It is no longer restricted by whether different object storage products themselves support multi-version control functions, so that the storage gateway can flexibly adapt to all object storage products, and the compatibility with various object storage products is greatly improved.

[0143] In an exemplary embodiment, the object storage management device also includes a reading module; the index file includes a position index area and a version index area, the position index area records the position information of each transmitted data object in the block device, and the version index area stores multiple index data; the receiving module 402 is also used to receive a data read request; the reading module is used to read data from the corresponding target block device according to the data read request; when the data is not read locally, the position index area is searched according to the position information of the target block device to determine whether the target data object corresponding to the target block device is a transmitted data object; when it is determined that the target data object is a transmitted data object, the unique storage name of the target data object is determined from the version index area; according to the determined unique storage name, data is obtained from the object storage system; and the obtained data is returned to the sender of the data read request.

[0144] In an exemplary embodiment, the object storage management device further includes a progress management module; the progress management module is used to record corresponding data object information in a progress status file for each data object to be transmitted if the current data object to be transmitted is uploaded.

[0145] In an exemplary embodiment, the progress management module is also used to stop uploading the data objects to be transmitted in each block device if a fault occurs; when it is determined that the fault is cleared, read the progress status file to obtain the data object information of the transmitted data objects; and continue to upload the data objects to be transmitted in each block device based on the obtained data object information.

[0146] In an exemplary embodiment, the index generation module 408 is further configured to generate an index object according to the index data; and upload the index object to the object storage system.

[0147] In an exemplary embodiment, the object storage management device also includes a deletion module, which is used to determine old version index data based on version information or time information carried by each index data in the index file; delete the data object corresponding to the old version index data in the object storage system; and delete the old version index object stored in the object storage system.

[0148] In an exemplary embodiment, the object storage management device also includes a recovery module, which is used to scan the index objects stored in the object storage system when local data is damaged; determine at least one latest index object based on the version information or time information carried by each index object; read the unique storage name of the data object to be recovered in at least one latest index object; obtain the target data from the object storage system based on the read unique storage name; and perform local data recovery based on the obtained target data.

[0149] Each module in the above object storage management device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in the gateway device in the form of hardware, or can be stored in the memory in the gateway device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0150] In an exemplary embodiment, a gateway device is provided, and the internal structure diagram of the gateway device can be shown in Figure 5. The gateway device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the gateway device is used to provide computing and control capabilities. The memory of the gateway device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the gateway device is used to exchange information between the processor and an external device. The communication interface of the gateway device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an object storage management method is implemented.

[0151] Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the gateway device to which the solution of the present application is applied. The specific gateway device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0152] In an exemplary embodiment, a gateway device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: receiving current data and writing the current data into a corresponding block device; the storage gateway includes multiple block devices, each block device divides data objects according to a continuous and fixed-length offset range; a unique storage name is set for the current data to constitute a data object to be transmitted; when a preset upload condition is met, the data object to be transmitted in each block device is uploaded to an object storage system; after the upload is completed, index data is generated according to the block device address and unique storage name of each transmitted data object; the index data carries version information or time information; and the index file is updated according to the index data.

[0153] In one embodiment, the processor further implements the following steps when executing the computer program: receiving a data read request; reading data from a corresponding target block device according to the data read request; if the data is not read locally, searching in a location index area according to location information of the target block device to determine whether a target data object corresponding to the target block device is a transmitted data object; if it is determined that the target data object is a transmitted data object, determining a unique storage name of the target data object from a version index area; obtaining data from an object storage system according to the determined unique storage name; and returning the obtained data to the sender of the data read request.

[0154] In one embodiment, when the processor executes the computer program, the following steps are further implemented: for each data object to be transmitted, if the current data object to be transmitted is uploaded, the corresponding data object information is recorded in the progress status file.

[0155] In one embodiment, the processor further implements the following steps when executing the computer program: if a fault occurs, stop uploading the data objects to be transmitted in each block device; when it is determined that the fault has been cleared, read the progress status file to obtain data object information of the transmitted data objects; and continue uploading the data objects to be transmitted in each block device based on the obtained data object information.

[0156] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: uploading the index object to the object storage system according to the index data.

[0157] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining old version index data based on version information or time information carried by each index data in the index file; deleting the data object corresponding to the old version index data in the object storage system; deleting the old version index object stored in the object storage system.

[0158] In one embodiment, the processor further implements the following steps when executing the computer program: in the event of local data corruption, scan each index object stored in the object storage system; determine at least one latest index object based on the version information or time information carried by each index object; read the unique storage name of the data object to be recovered in at least one latest index object; obtain target data from the object storage system based on the read unique storage name; and perform local data recovery based on the obtained target data.

[0159] In one embodiment, as shown in FIG. 1 , a storage system is provided, the storage system comprising: a storage gateway 10 and an object storage system 20; wherein: the storage gateway 10 is used to perform the following steps: receiving current data and writing the current data into a corresponding block device; the storage gateway 10 comprises a plurality of block devices, each block device dividing data objects according to a continuous and fixed-length offset range; setting a unique storage name for the current data to constitute a data object to be transmitted; uploading the data object to be transmitted in each block device to the object storage system 20 when a preset upload condition is met; after the upload is completed, generating index data according to the block device address and the unique storage name of each transmitted data object; the index data carries version information or time information; and updating the index file according to the index data.

[0160] In one embodiment, the index file includes a location index area and a version index area, the location index area records the location information of each transmitted data object in the block device, and the version index area stores multiple index data; the storage gateway 10 is used to perform the following steps: receive a data read request; read data from the corresponding target block device according to the data read request; if the data is not read locally, search in the location index area according to the location information of the target block device to determine whether the target data object corresponding to the target block device is a transmitted data object; if it is determined that the target data object is a transmitted data object, determine the unique storage name of the target data object from the version index area; obtain data from the object storage system 20 according to the determined unique storage name; and return the obtained data to the sender of the data read request.

[0161] In one embodiment, the storage gateway 10 is used to perform the following steps: for each data object to be transmitted, if the current data object to be transmitted is uploaded, the corresponding data object information is recorded in the progress status file.

[0162] In one embodiment, the storage gateway 10 is used to perform the following steps: if a fault occurs, stop uploading the data objects to be transmitted in each block device; when it is determined that the fault is cleared, read the progress status file to obtain the data object information of the transmitted data objects; based on the obtained data object information, continue to upload the data objects to be transmitted in each block device.

[0163] In one embodiment, the storage gateway 10 is used to perform the following steps: generating an index object according to the index data; and uploading the index object to the object storage system 20.

[0164] In one embodiment, the storage gateway 10 is used to perform the following steps: determine the old version index data based on the version information or time information carried by each index data in the index file; delete the data object corresponding to the old version index data in the object storage system 20; delete the old version index object stored in the object storage system 20.

[0165] In one embodiment, the storage gateway 10 is used to perform the following steps: in the event of local data corruption, scan each index object stored in the object storage system 20; determine at least one latest index object based on the version information or time information carried by each index object; read the unique storage name of the data object to be restored in at least one latest index object; obtain the target data from the object storage system 20 based on the read unique storage name; and perform local data recovery based on the obtained target data.

[0166] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0167] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0168] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0169] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An object storage management method, characterized in that: The method is applied to a storage gateway, and the method includes: Receive current data and write the current data into a corresponding block device; the storage gateway includes multiple block devices, each block device divides data objects according to a continuous and fixed-length offset range; Setting a unique storage name for the current data to form a data object to be transmitted; When the preset upload conditions are met, the data objects to be transmitted in each block device are uploaded to the object storage system; After the upload is completed, index data is generated according to the block device address and unique storage name of each uploaded data object; the index data carries version information or time information; The index file is updated according to the index data.

2. The method according to claim 1, characterized in that The index file includes a position index area and a version index area, the position index area records the position information of each transmitted data object in the block device, and the version index area stores a plurality of index data; the method further includes: Receive a data read request; Reading data from a corresponding target block device according to the data read request; In the case where no data is read locally, searching the location index area according to the location information of the target block device to determine whether the target data object corresponding to the target block device is a transmitted data object; In the case where it is determined that the target data object is a transmitted data object, determining a unique storage name of the target data object from the version index area; Retrieving data from the object storage system according to the determined unique storage name; The acquired data is returned to the sender of the data reading request.

3. The method according to claim 1, characterized in that After uploading the data objects to be transmitted in each block device to the object storage system, the method further includes: For each data object to be transmitted, if the current data object to be transmitted is uploaded, the corresponding data object information is recorded in the progress status file.

4. The method according to claim 3, characterized in that The method further comprises: If a failure occurs, the uploading of the data objects to be transmitted in each device will be stopped; When it is determined that the fault has been cleared, reading the progress status file to obtain data object information of the transmitted data object; According to the acquired data object information, continue to upload the data objects to be transmitted in each block device.

5. The method according to any one of claims 1 to 4, characterized in that: After updating the index file according to the index data, the method further includes: Generate an index object according to the index data; Upload the index object to the object storage system.

6. The method according to claim 5, characterized in that The method further comprises: Determine the old version index data according to the version information or time information carried by each index data in the index file; Deleting the data object corresponding to the old version index data in the object storage system; The old version index object stored in the object storage system is deleted.

7. The method according to claim 6, characterized in that The method further comprises: In case of local data corruption, scanning each index object stored in the object storage system; Determine at least one latest index object according to the version information or time information carried by each index object; Reading a unique storage name of the data object to be restored in the at least one latest index object; Obtain target data from the object storage system according to the read unique storage name; Perform local data recovery based on the acquired target data.

8. An object storage management device, characterized in that: The device is applied to a storage gateway, and the device includes: A receiving module, used to receive current data and write the current data into a corresponding block device; the storage gateway includes a plurality of block devices, each of which divides data objects according to a continuous and fixed-length offset range; A naming module, used to set a unique storage name for the current data to constitute a data object to be transmitted; The upload module is used to upload the data objects to be transmitted in each block device to the object storage system when the preset upload conditions are met; The index generation module is used to generate index data according to the block device address and unique storage name of each uploaded data object after the upload is completed; the index data carries version information or time information; and the index file is updated according to the index data.

9. A gateway device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A storage system, characterized in that: The storage system includes: a storage gateway and an object storage system; wherein: The storage gateway is used to perform the steps of the method according to any one of claims 1 to 7.

Images