Object multi-version intercommunication access method, system, device and medium

Abstract

The present invention relates to a method, system, device and medium for intercommunication access of multiple versions of an object. The method comprises: obtaining an operation request of an s3 gateway, and then performing an operation based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least comprises: an overwrite write operation of the s3 gateway, a delete operation of the s3 gateway and a historical version operation of the s3 gateway; obtaining an IO request of other gateways, then identifying an identifier of a corresponding file, searching for corresponding file metadata in the current object set, and performing a read operation and / or a write operation of the other gateways. The present invention enables the multi-version function of 3s objects to function normally and without performance degradation during intercommunication access of multiple protocols such as NFS, SAMBA, FTP and the like.

Description

Technical Field

[0001] The present invention relates to the field of distributed S3 object storage technology, and in particular to a method, system, device and medium for intercommunication access of multiple versions of an object. Background Art

[0002] S3 object storage [Amazon Simple Storage Service (Amazon S3)] is an object storage service that provides object storage through an interface based on a RESTFUL API. It is a technology for storing and managing data in an unstructured format. In actual usage scenarios, users usually need to use different protocols to access the same data. For example, data written through S3 needs to be accessed through protocols such as NFS (Network File System), SAMBA (a free software that implements the SMB protocol on Linux and UNIX systems), and FTP (File Transfer Protocol). In the method of the present invention, the method of accessing the same data using multiple different protocols is called interoperable access.

[0003] The multi-version feature of s3 object storage allows objects in a bucket to have multiple versions, so that the change history of the object can be tracked. This feature is very useful for application scenarios that require data recovery, auditing or compliance requirements. After the multi-version feature of objects is enabled, multi-protocol interoperability becomes a more challenging point. For example, after s3 enables the multi-version feature, an object is overwritten. In the s3 multi-version interface query, the historical version records must be viewed and the historical version data must be retrieved. At the same time, when accessing the object just written to s3 through protocols such as NFS, SAMBA, and FTP, it is required that only the latest written data can be seen, and the old data cannot be accessed. That is, historical data is not visible to other non-s3 protocols such as NFS, SAMBA, and FTP. Among the few existing s3 object storage products that support multi-protocol interoperability, they often cannot support the multi-version feature of objects and interoperability with other protocols.

[0004] Therefore, how to invent a multi-version function for S3 object storage products and other protocol intercommunication technologies is an urgent problem to be solved. Summary of the invention

[0005] The present invention provides a method, system, device and medium for intercommunication access of multiple versions of an object, so as to solve the problem in the prior art that special conversion is required between s3 objects and various protocols such as NFS, SAMBA, FTP, etc., and direct intercommunication access cannot be achieved.

[0006] To achieve the above-mentioned purpose, in a first aspect, the present invention relates to a method for interoperable access of multiple versions of an object, which is used for an S3 object in a distributed storage system, comprising:

[0007] Obtaining an operation request of an s3 gateway, and then performing an operation based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least includes: an overwrite operation of the s3 gateway, a delete operation of the s3 gateway, and a historical version operation of the s3 gateway;

[0008] The overwrite operation of the s3 gateway is as follows: identify the object to be written by s3, check whether the s3 object requested this time exists in the current object set, if not, write it as a new object; if it exists, move the metadata of the old object from the current object set to the historical deletion object set, record the deletion time, and generate a new globally unique object ID, locate the location where the object needs to be written through the globally unique object ID, and send the content to be written this time to the write location to complete the data writing operation;

[0009] The deletion operation of the s3 gateway is: moving the s3 object to be deleted from the current object set to the historical deletion object set and recording the deletion time, and creating a deletion mark in the historical deletion mark table;

[0010] And the historical version listing operation of the s3 gateway is: listing the current version objects from the current object set, listing the historical version objects from the historical deletion object set, and listing the deletion marks from the historical deletion mark table;

[0011] Obtain IO requests from other gateways, identify the identifier of the corresponding file, search for the corresponding file metadata in the current object set, and perform read operations and / or write operations on the other gateways, wherein the other gateways include one or more of an NFS gateway, a SAMBA gateway, and an FTP gateway.

[0012] To achieve the above-mentioned object, in a second aspect, the present invention relates to an object multi-version intercommunication access system for distributed storage system s3 objects, comprising: an operation module of an s3 gateway and other gateway access modules;

[0013] The operation module of the s3 gateway is used to obtain an operation request of the s3 gateway, and then perform an operation based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least includes: an overwrite operation of the s3 gateway, a delete operation of the s3 gateway, and a historical version operation of the s3 gateway;

[0014] The overwrite operation of the s3 gateway is as follows: identify the object to be written by s3, check whether the s3 object requested this time exists in the current object set, if not, write it as a new object; if it exists, move the metadata of the old object from the current object set to the historical deletion object set, record the deletion time, and generate a new globally unique object ID, locate the location where the object needs to be written through the globally unique object ID, and send the content to be written this time to the write location to complete the data writing operation;

[0015] The deletion operation of the s3 gateway is: moving the s3 object to be deleted from the current object set to the historical deletion object set and recording the deletion time, and creating a deletion mark in the historical deletion mark table;

[0016] And the historical version listing operation of the s3 gateway is: listing the current version objects from the current object set, listing the historical version objects from the historical deletion object set, and listing the deletion marks from the historical deletion mark table;

[0017] The other gateway access module is used to obtain IO requests from other gateways, identify the identifier of the corresponding file, search for the corresponding file metadata in the current object set, and perform read operations and / or write operations on the other gateways, wherein the other gateways include one or more of NFS gateways, SAMBA gateways and FTP gateways.

[0018] To achieve the above objectives, in a third aspect, the present invention also relates to an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method for multi-version interoperable access to objects when executing the computer program.

[0019] To achieve the above-mentioned purpose, in a fourth aspect, the present invention also relates to a computer-readable storage medium, in which instructions are stored, and when the instructions are run, the above-mentioned method for interoperable access to multiple versions of an object is executed.

[0020] The object multi-version intercommunication access method, system, device and medium disclosed in the present invention have the following beneficial effects compared with the prior art:

[0021] The current version of s3 multi-version is stored in the current object collection. Its metadata is the same as the file metadata created by protocols such as NFS, SAMBA, and FTP. Gateways such as NFS, SAMBA, and FTP can only access the current object collection. Therefore, after the s3 multi-version overwrite, gateways such as NFS, SAMBA, and FTP can access the latest written data, while the historical object metadata is transferred to the historical deletion object collection, which can only be accessed by the s3 gateway. Similarly, because it is the same metadata, files written through gateways such as NFS, SAMBA, and FTP can also be immediately reflected in the current version of s3, and the latest changed data can be read through the s3 gateway. Because the underlying metadata is the same, the data access method is the same, and different protocols can directly access it through different gateways, but the deleted historical object metadata cannot be accessed by the above non-s3 protocols.

[0022] Therefore, by distinguishing and managing the object metadata currently in use and the deleted historical object metadata, other non-s3 protocols such as NFS, SAMBA, FTP, etc. can always access the current latest version of the object metadata when accessing s3 object multi-version data. The present invention does not require any conversion, so there is no performance loss in the intercommunication access between different protocols, the functions are normal, and the performance is not reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A method flow chart of a method for interoperating multiple versions of an object in a first embodiment of the present invention;

[0024] Figure 2 This is a flowchart of an overwrite write operation example of an s3 gateway in a method for interoperable access to multiple versions of an object in Embodiment 1 of the present invention;

[0025] Figure 3 This is a flowchart of a deletion operation example of an S3 gateway in a method for intercommunication and accessing multiple versions of an object in Embodiment 1 of the present invention;

[0026] Figure 4 This is a flowchart of an example of the historical version enumeration operation of an S3 gateway in a method for intercommunication and accessing multiple versions of an object in Embodiment 1 of the present invention;

[0027] Figure 5 This is a flowchart of an IO operation example of other gateways of a method for interoperating access to multiple versions of an object in Embodiment 1 of the present invention;

[0028] Figure 6 This is a flowchart of an operation example of retrieving historical version data of a method for interoperating multiple versions of an object in Embodiment 1 of the present invention;

[0029] Figure 7This is a flowchart of an operation example of deleting historical version data of a method for interoperating multiple versions of an object in Embodiment 1 of the present invention;

[0030] Figure 8 It is a structural diagram of a multi-version intercommunication access system for objects in Embodiment 2 of the present invention;

[0031] Fig. 9 It is a structural schematic diagram of an electronic device in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0032] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0033] Embodiment 1

[0034] A method for interoperating multiple versions of an object. Figure 1-Figure 6 , implemented by an electronic device with a processor, and can be implemented for personal computers, smart terminals, local area networks, servers, etc., for s3 objects in a distributed storage system. The distributed storage system includes a storage network, an S3 gateway, other gateways, a multi-node metadata manager, a container and object collection, a file location generator, a storage node, a hard disk on the node, and a local file system after the hard disk is formatted. Among them, other gateways can be one or more of NFS gateways, SAMBA gateways, and FTP gateways. The multi-node metadata manager is responsible for generating a globally unique ID for the metadata of each object and file. The file location generator calculates the storage location based on the globally unique ID and performs corresponding IO operations.

[0035] like Figure 1 As shown, the following steps are included: S101 to S102.

[0036] S101: Obtain an operation request of the s3 gateway, and then perform an operation based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least includes: an overwrite operation of the s3 gateway, a delete operation of the s3 gateway, and a historical version operation of the s3 gateway;

[0037] Among them, the overwrite write operation of the s3 gateway is:

[0038] Identify the object written to s3;

[0039] Check whether the S3 object requested this time exists in the current object collection. If not, write it as a new object. If it does, move the metadata of the old object from the current object collection to the historical deletion object collection, record the deletion time, and generate a new globally unique object ID. Use the globally unique object ID to locate the location where the object needs to be written, and send the content to be written this time to the write location to complete the data writing operation.

[0040] The following examples are used to illustrate the overwrite operation of the s3 gateway. Figure 2 shown.

[0041] Step 11: Check whether the object multi-version function of the s3 bucket is enabled. If it is not enabled, return directly and handle it according to the normal overwrite process;

[0042] Step 12: Parse the s3 request, identify the s3 object, and check in the multi-node metadata manager whether the s3 object requested this time exists; if the s3 object does not exist, write it as a new object; if the s3 object exists, move the metadata of the old object from the current object set to the historical deletion object set, and add the deletion time field to the historical deletion object set;

[0043] Step 13: Generate a new globally unique object ID; the location generator locates the location where the object needs to be written through the ID;

[0044] Step 14: Send the content to be written this time to the write location obtained in step 3 to complete the data writing operation; if the writing fails, the overwrite write request processing fails, and the rollback process is performed, that is, the metadata record moved to the historical deletion object set in step 2 is moved to the current object set; if the writing is successful, the next step of metadata preservation is performed;

[0045] Step 15: Using the globally unique object ID as the key, store the time of this write, object size, MD5 value of this data content and other attributes into the current object collection in the multi-node metadata manager;

[0046] Step 16: The write request processing is completed.

[0047] The deletion operation of the s3 gateway is: move the s3 object to be deleted from the current object set to the historical deletion object set and record the deletion time, and create a deletion mark in the historical deletion mark table.

[0048] In some embodiments, the deletion operation of the 3S gateway also includes, if the object to be deleted is the first deletion marker, then after deleting the first deletion marker, checking whether there is still a first deletion marker, if not, moving the last deleted object in the historical deletion object set to the current object set; if there is still a first deletion marker, ending the operation.

[0049] The following examples are used to illustrate the deletion operation of the s3 gateway. Figure 3 shown.

[0050] Step 21 checks whether the object multi-version function of the s3 bucket is enabled. If it is not enabled, it returns directly and handles it according to the normal deletion process;

[0051] Step 22 parses the s3 request, identifies the s3 object, and if the object to be deleted exists, moves the old object to the historical deleted object set, increases the deletion time, and finally creates a deletion marker in the historical deleted object set;

[0052] Step 23: If the object to be deleted is a deletion mark, after deleting the deletion mark, check whether the object still has a deletion mark. If not, move the object that was deleted most recently in the historical deletion object set to the current object set. If there is still a deletion mark, return directly.

[0053] Step 24: The deletion request processing is completed.

[0054] The historical version enumeration operation of the s3 gateway is: list the current version objects from the current object collection, list the historical version objects from the historical deletion object collection, and list the deletion markers from the historical deletion marker table.

[0055] In this implementation, historical version objects are listed from the historical deletion object set, and deletion object tags are listed from the historical deletion mark table. Specifically, the historical deletion object set is listed. If it is a deletion object with a deletion mark in the historical deletion mark table, the listed file size is set to 0. If it is a historical object without a deletion mark, the historical object metadata is returned in accordance with the s3 protocol requirements.

[0056] The following examples are used to illustrate the historical version listing operation of the s3 gateway. Figure 4 shown.

[0057] Step 31 checks whether the object multi-version function of the s3 bucket is enabled. If it is not enabled, it returns directly and processes according to the normal deletion process;

[0058] Step 32 parses the s3 request, identifies the s3 object, and if the object to be deleted exists, moves the old object to the historical deleted object set, increases the deletion time, and finally creates a deletion marker in the historical deleted object set;

[0059] Step 33: If the object to be deleted is a deletion mark, after deleting the deletion mark, check whether the object still has a deletion mark. If not, move the object deleted most recently in the historical deletion object set to the current object set; if there is still a deletion mark, return directly;

[0060] Step 34: This deletion request processing is completed.

[0061] S102: Obtain IO requests from other gateways, identify the identifier of the corresponding file, search for the corresponding file metadata in the current object set, and perform read operations and / or write operations on other gateways, where the other gateways include one or more of NFS gateways, SAMBA gateways, and FTP gateways.

[0062] The following examples are used to illustrate the IO operations of other gateways. Figure 5 shown.

[0063] Step 41: Each gateway parses and processes the corresponding protocol request: the NFS gateway processes the NFS protocol, the SAMBA gateway processes the SAMBA protocol, and the FTP gateway processes the FTP protocol;

[0064] Step 42: After receiving the file access request of the respective protocols, the NFS gateway, the SAMBA gateway, and the FTP gateway parse and identify the identifier of the corresponding file, and search for the corresponding file metadata in the current object set of the multi-node metadata manager to obtain the file ID, file size, and other information;

[0065] Step 43: The location generator calculates the path of the file in the local file system according to the file ID, and performs corresponding data access according to the request type; if it is a write, the data is written to the local file system, and after the write is successful, the metadata such as the file size, modification time, etc. are updated; if it is a read, the data of the local file system is returned to the client;

[0066] Step 44: End and return.

[0067] In some embodiments, the operation of the s3 gateway also includes: retrieving historical version data operations, obtaining the historical version number ID input by the s3 client, retrieving the file ID and file size of the historical version data from the historical deletion object set according to the historical version number ID, calculating the local file system path according to the file ID, and then reading data from the local file system path according to the file size, and returning the read data to the s3 client.

[0068] The following examples are used to illustrate the operation of retrieving historical version data. Figure 6 shown.

[0069] Step 51 parses the s3 request and obtains the retrieved historical version number ID;

[0070] Step 52 obtains the file ID and file size from the historical deletion object set in the multi-node metadata manager;

[0071] Step 53: The location generator calculates the path of the file in the local file system according to the file ID, and then reads the data from the local file system according to the file size;

[0072] Step 54 returns the read data to the client;

[0073] Step 55 ends and returns.

[0074] In some embodiments, the operation of the s3 gateway also includes deleting historical version data:

[0075] Get the retrieved historical version number ID;

[0076] Get the file ID and file size from the historical deletion object collection in the multi-node metadata manager;

[0077] The location generator calculates the path of the file in the local file system based on the file ID and deletes the local file system data;

[0078] Delete history deletes the metadata of object history versions recorded in the object collection.

[0079] The following examples are used to illustrate the operation of deleting historical version data. Figure 7 shown.

[0080] Step 61 parses the s3 request and obtains the retrieved historical version number ID;

[0081] Step 62 obtains the file ID and file size from the historical deletion object set in the multi-node metadata manager;

[0082] Step 63: The location generator calculates the path of the file in the local file system according to the file ID and deletes the local file system data;

[0083] Step 64 deletes the object history version metadata recorded in the history deletion object set;

[0084] Step 65 ends and returns.

[0085] In summary, the object multi-version intercommunication access method proposed by the present invention, the current version of s3 multi-version is stored in the current object set, its metadata and the file metadata created by protocols such as NFS, SAMBA, FTP, etc. are the same, and gateways such as NFS gateway, SAMBA gateway, FTP gateway, etc. can only access the current object set, so after the s3 multi-version overwrite, the NFS gateway, SAMBA gateway, FTP gateway, etc. gateways can access the latest written data, and the historical object metadata is transferred to the historical deletion object set, which can only be accessed by the s3 gateway. Similarly, because it is the same metadata, the files written by gateways such as NFS gateway, SAMBA gateway, FTP gateway, etc. can also be immediately reflected in the current version of s3, and the latest changed data can be read through the s3 gateway. Because the underlying metadata is the same, the data access method is the same, and different protocols can be directly accessed through different gateways without any conversion, so there is no performance loss in intercommunication access between different protocols. .

[0086] Embodiment 2

[0087] An object multi-version intercommunication access system is used for distributed storage system S3 objects, which is implemented by electronic equipment hardware with a processor and can be implemented for personal computers, smart terminals, local area networks, servers, etc. Figure 8 , including: an operation module 81 of the s3 gateway and other gateway access modules 82;

[0088] The operation module 81 of the s3 gateway is used to obtain the operation request of the s3 gateway, and then perform operations based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least includes: the overwrite operation of the s3 gateway, the deletion operation of the s3 gateway, and the historical version operation of the s3 gateway;

[0089] The overwrite operation of the s3 gateway is as follows: identify the object to be written by s3, check whether the s3 object requested this time exists in the current object set, if not, write it as a new object; if it exists, move the metadata of the old object from the current object set to the historical deletion object set, record the deletion time, and generate a new globally unique object ID, locate the location where the object needs to be written through the globally unique object ID, and send the content to be written this time to the write location to complete the data writing operation;

[0090] The deletion operation of the s3 gateway is: move the s3 object to be deleted from the current object set to the historical deletion object set and record the deletion time, and create a deletion mark in the historical deletion mark table;

[0091] And the historical version listing operation of the s3 gateway is: list the current version object from the current object set, list the historical version object from the historical deletion object set, and list the deletion mark from the historical deletion mark table;

[0092] The other gateway access module 82 is used to obtain IO requests from other gateways, identify the identifier of the corresponding file, search for the corresponding file metadata in the current object set, and perform read operations and / or write operations on other gateways, where other gateways include one or more of NFS gateways, SAMBA gateways, and FTP gateways.

[0093] In some embodiments, the operation of the s3 gateway further includes a deletion history mark table operation, and the deletion history mark table operation is specifically performed as follows: receiving a request for the history mark table operation, removing the corresponding deletion object mark from the history mark table, and if all the deletion marks of the s3 object are cleared, selecting the object with the longest deletion time from the history delete object set, and moving it to the current object set;

[0094] In some embodiments, the operation of the s3 gateway further includes an operation of retrieving historical version data, and the operation of retrieving historical version data is specifically performed as follows: obtaining the historical version number ID input by the s3 client, retrieving the file ID and file size of the historical version data from the historical deletion object set according to the historical version number ID, calculating the local file system path according to the file ID, and then reading data from the local file system path according to the file size, and returning the read data to the s3 client;

[0095] In some embodiments, the operation of the s3 gateway also includes an operation of deleting historical version data. The specific operation of deleting historical version data is as follows: obtaining the retrieved historical version number ID; obtaining the file ID and file size from the historical deletion object set in the multi-node metadata manager; calculating the file path in the local file system based on the file ID, and deleting the local file system data; deleting the object historical version metadata recorded in the historical deletion object set.

[0096] In some embodiments, the deletion operation of the 3S gateway also includes, if the object to be deleted is the first deletion marker, then after deleting the first deletion marker, checking whether there is still a first deletion marker, if not, moving the last deleted object in the historical deletion object set to the current object set; if there is still a first deletion marker, ending the operation.

[0097] In some embodiments, historical version objects are listed from a historical deletion object set, and deletion object marks are listed from a historical deletion mark table. Specifically, the historical deletion object set is listed. If it is a deletion object with a deletion mark in the historical deletion mark table, the listed file size is set to 0. If it is a historical object without a deletion mark, the historical object metadata is returned in accordance with the s3 protocol requirements.

[0098] The implementation process, method and effect of the object multi-version intercommunication access system of this embodiment are the same as those of the object multi-version intercommunication access method described in the first embodiment, and will not be repeated here.

[0099] Embodiment 3

[0100] like Fig. 9 As shown, the present embodiment relates to an electronic device including at least one processor and a memory connected to the at least one processor in communication. The memory stores a computer program that can be run by at least one processor, and the computer program is executed by at least one processor so that at least one processor can execute a method for multi-version intercommunication access to an object in Example 1, and realize the corresponding beneficial effects of a method for multi-version intercommunication access to an object, which will not be repeated here. The electronic device provided in the present embodiment can be a personal computer, such as a desktop computer, an all-in-one computer, a laptop computer, a tablet computer, etc., and can also be a terminal device such as a mobile phone, a wearable device, a handheld computer, etc. In the present embodiment, the electronic device is a flight control computer. The electronic device is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0101] The components of the electronic device 3 may include, but are not limited to: the at least one processor 4 mentioned above, the at least one memory 5 mentioned above, and a bus 6 connecting different system components (including the memory 5 and the processor 4).

[0102] The bus 6 includes a data bus, an address bus and a control bus.

[0103] The memory 5 may include a volatile memory, such as a random access memory (RAM) 51 and / or a cache memory 52 , and may further include a read-only memory (ROM) 53 .

[0104] The memory 5 may also include a program / utility 55 having a set (at least one) of program modules 54, such program modules 54 including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0105] The processor 4 executes various functional applications and data processing by running the computer programs stored in the memory 5, such as the above-mentioned object multi-version intercommunication access method.

[0106] The electronic device 3 may also communicate with one or more external devices 7 (e.g., keyboards, pointing devices, etc.). Such communication may be performed via an input / output (I / O) interface 8. Furthermore, the electronic device 3 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 9. Figure 5 As shown, the network adapter 9 communicates with other modules of the electronic device 3 via the bus 6. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 3, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0107] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be embodied.

[0108] Embodiment 4

[0109] The present invention relates to a computer-readable storage medium, which stores instructions. When the instructions are executed, a method for intercommunication accessing multiple versions of an object is executed. The implementation process, method and effect of the method are the same as those of the method for intercommunication accessing multiple versions of an object described in Example 1, and will not be repeated here.

[0110] It should be noted that, in this article, the terms include, contain or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence includes a ... does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0111] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for interoperating multiple versions of an object, characterized in that: S3 objects for distributed storage systems, including: Obtaining an operation request of an s3 gateway, and then performing an operation based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least includes: an overwrite operation of the s3 gateway, a delete operation of the s3 gateway, and a historical version operation of the s3 gateway; The overwrite operation of the s3 gateway is as follows: identify the object to be written by s3, check whether the s3 object requested this time exists in the current object set, if not, write it as a new object; if it exists, move the metadata of the old object from the current object set to the historical deletion object set, record the deletion time, and generate a new globally unique object ID, locate the location where the object needs to be written through the globally unique object ID, and send the content to be written this time to the write location to complete the data writing operation; The deletion operation of the s3 gateway is: moving the s3 object to be deleted from the current object set to the historical deletion object set and recording the deletion time, and creating a deletion mark in the historical deletion mark table; And the historical version listing operation of the s3 gateway is: listing the current version objects from the current object set, listing the historical version objects from the historical deletion object set, and listing the deletion marks from the historical deletion mark table; Obtain IO requests from other gateways, identify the identifier of the corresponding file, search for the corresponding file metadata in the current object set, and perform read operations and / or write operations on the other gateways, wherein the other gateways include one or more of an NFS gateway, a SAMBA gateway, and an FTP gateway.

2. The object multi-version intercommunication access method according to claim 1, characterized in that: The operation of the s3 gateway also includes: deleting the historical deletion mark table operation, receiving a request for the historical deletion mark table operation, removing the corresponding deletion object mark from the historical deletion mark table, and if all the deletion marks of the s3 object are cleared, selecting the object with the longest deletion time from the historical deletion object set and moving it to the current object set.

3. The object multi-version intercommunication access method according to claim 2, characterized in that: The operation of the s3 gateway also includes: retrieving historical version data, obtaining the historical version number ID input by the s3 client, retrieving the file ID and file size of the historical version data from the historical deletion object set according to the historical version number ID, calculating the local file system path according to the file ID, and then reading data from the local file system path according to the file size, and returning the read data to the s3 client.

4. The object multi-version intercommunication access method according to claim 3, characterized in that: The operation of the s3 gateway also includes deleting historical version data: Get the retrieved historical version number ID; Obtaining a file ID and a file size from the historical deletion object set in the multi-node metadata manager; The location generator calculates the path of the file in the local file system based on the file ID and deletes the local file system data; Delete the object history version metadata recorded in the history deletion object set.

5. The object multi-version intercommunication access method according to claim 3, characterized in that: The deletion operation of the 3S gateway also includes, if the object to be deleted is a first deletion marker, then after deleting the first deletion marker, checking whether there is still a first deletion marker, if not, moving the object most recently deleted in the historical deletion object set to the current object set; if the first deletion marker still exists, ending the operation.

6. The object multi-version intercommunication access method according to claim 3, characterized in that: The method of listing historical version objects from the historical deletion object set and listing deletion object marks from the historical deletion mark table is specifically as follows: listing the historical deletion object set, if it is a deletion object with a deletion mark in the historical deletion mark table, setting the listed file size to 0, and if it is a historical object without a deletion mark, returning the historical object metadata in accordance with the s3 protocol requirements.

7. An object multi-version intercommunication access system, characterized in that: Used for distributed storage system s3 objects, including: s3 gateway operation module and other gateway access modules; The operation module of the s3 gateway is used to obtain an operation request of the s3 gateway, and then perform an operation based on the operation request of the s3 gateway, wherein the operation of the s3 gateway at least includes: an overwrite operation of the s3 gateway, a delete operation of the s3 gateway, and a historical version operation of the s3 gateway; The overwrite operation of the s3 gateway is as follows: identify the object to be written by s3, check whether the s3 object requested this time exists in the current object set, if not, write it as a new object; if it exists, move the metadata of the old object from the current object set to the historical deletion object set, record the deletion time, and generate a new globally unique object ID, locate the location where the object needs to be written through the globally unique object ID, and send the content to be written this time to the write location to complete the data writing operation; The deletion operation of the s3 gateway is: moving the s3 object to be deleted from the current object set to the historical deletion object set and recording the deletion time, and creating a deletion mark in the historical deletion mark table; And the historical version listing operation of the s3 gateway is: listing the current version objects from the current object set, listing the historical version objects from the historical deletion object set, and listing the deletion marks from the historical deletion mark table; The other gateway access module is used to obtain IO requests from other gateways, identify the identifier of the corresponding file, search for the corresponding file metadata in the current object set, and perform read operations and / or write operations on the other gateways, wherein the other gateways include one or more of NFS gateways, SAMBA gateways and FTP gateways.

8. The object multi-version intercommunication access system according to claim 7, characterized in that: The operation of the s3 gateway also includes deleting the historical deletion mark table operation, and the specific execution of the deleting the historical deletion mark table operation is: receiving a request for the historical deletion mark table operation, removing the corresponding deletion object mark from the historical deletion mark table, and if all the deletion marks of the s3 object are cleared, selecting the object with the longest deletion time from the historical deletion object set, and moving it to the current object set; The operation of the s3 gateway also includes an operation of retrieving historical version data. The operation of retrieving historical version data is specifically performed as follows: obtaining the historical version number ID input by the s3 client, retrieving the file ID and file size of the historical version data from the historical deletion object set according to the historical version number ID, calculating the local file system path according to the file ID, and then reading data from the local file system path according to the file size, and returning the read data to the s3 client; Or the operation of the s3 gateway also includes an operation of deleting historical version data, and executing the operation of deleting historical version data is specifically as follows: obtaining the retrieved historical version number ID; obtaining the file ID and file size from the historical deletion object set in the multi-node metadata manager; calculating its path in the local file system based on the file ID, and deleting the local file system data; deleting the object historical version metadata recorded in the historical deletion object set.

9. The object multi-version intercommunication access system according to claim 7, characterized in that: The deletion operation of the 3S gateway also includes, if the object to be deleted is a first deletion marker, then after deleting the first deletion marker, checking whether there is still a first deletion marker, if not, moving the object most recently deleted in the historical deletion object set to the current object set; if the first deletion marker still exists, ending the operation.

10. The object multi-version intercommunication access system according to claim 7, characterized in that: The method of listing historical version objects from the historical deletion object set and listing deletion object marks from the historical deletion mark table is as follows: listing the historical deletion object set, if it is a deletion object with a deletion mark in the historical deletion mark table, setting the listed file size to 0, and if it is a historical object without a deletion mark, returning the historical object metadata in accordance with the s3 protocol requirements.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements a method for multi-version intercommunication access to an object as described in any one of claims 1 to 7.

12. A computer-readable storage medium, characterized in that: The storage medium stores instructions, which, when executed, execute a method for interoperable access of multiple versions of an object as described in any one of claims 1 to 7.

Images