A data access method and system for a storage gateway system

Abstract

The present invention discloses a data access method and system for a storage gateway system. The method includes: performing metadata operations through a file protocol module in the storage gateway system and a cached file subsystem in the storage gateway system, where the cached file subsystem includes a metadata cache module and a data cache module; and performing data access through the file protocol module and the cached file subsystem based on the metadata in the metadata cache module after the metadata operations. By using this method, the data access speed of the storage gateway can be accelerated.

Description

Technical Field

[0001] The present invention relates to the field of file storage technology, and more particularly to a data access method and system for a storage gateway system. Background Art

[0002] With the advancement of information technology, enterprises are experiencing explosive growth in unstructured data. Legacy business systems often rely on NAS (Network Attached Storage) or SAN (Storage Area Network) for back-end storage of massive amounts of unstructured data. However, these legacy file storage devices suffer from poor scalability and maintainability. Therefore, there is an urgent need for advanced unstructured data management systems to manage this ever-increasing volume of unstructured data.

[0003] Most unstructured data management systems on the market use proprietary protocols that differ significantly from traditional file storage protocols. This makes upgrading legacy business systems difficult and time-consuming. Therefore, existing technologies typically connect legacy storage devices to the unstructured data management system's storage gateway system. The storage gateway system then handles business system I / O using traditional protocols, performs protocol conversion, and performs asynchronous online migration of legacy data.

[0004] The existing technology requires first using the network protocol to write to the storage gateway system, and then the storage gateway system writes to the original NAS device through the NAS protocol. This causes the business system IO to be processed by the network protocol twice, thereby significantly reducing the access performance of the business system. Summary of the Invention

[0005] The present invention provides a data access method and system for a storage gateway system, so as to solve the problem in the prior art that the storage gateway system performs network protocol forwarding multiple times, resulting in a significant decrease in data access performance, and can accelerate the data access speed of the storage gateway.

[0006] According to one aspect of the present invention, a data access method of a storage gateway system is provided, comprising:

[0007] Perform metadata operations through a file protocol module in the storage gateway system and a cache file subsystem in the storage gateway system, wherein the cache file subsystem includes a metadata cache module and a data cache module;

[0008] Based on the metadata in the metadata cache module after the metadata operation, data access is performed through the file protocol module and the cache file subsystem.

[0009] According to another aspect of the present invention, there is provided a storage gateway system, which includes a file protocol module and a cached file subsystem. The protocol module is connected to the cached file subsystem.

[0010] The cached file subsystem includes a synchronization module, a metadata cache module, and a data cache module. The synchronization module is respectively connected to the metadata cache module and the data cache module.

[0011] Metadata operations are performed through the file protocol module in the storage gateway system and the cached file subsystem in the storage gateway system.

[0012] Based on the metadata in the metadata cache module after the metadata operation, data access is performed through the file protocol module and the cached file subsystem.

[0013] The technical solution of the embodiment of the present invention solves the problem of multiple protocol forwards required during data access in the prior art through the metadata cache module and the data cache module in the cached file subsystem, achieving the beneficial effect of accelerating data access of the storage gateway.

[0014] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0016] Figure 1 It is a schematic flowchart of a data access method for a storage gateway system provided by Embodiment 1 of the present invention.

[0017] Figure 2 It is a schematic diagram of a metadata cache tree structure in a data access method for a storage gateway system provided by Embodiment 1 of the present invention.

[0018] Figure 3 It is a flowchart example of metadata operations in a data access method for a storage gateway system provided by Embodiment 2 of the present invention.

[0019] Figure 4 It is a flowchart example of a data read operation in a data access method for a storage gateway system provided by Embodiment 2 of the present invention.

[0020] Figure 5 It is a flowchart example of the data writing operation in a data access method of a storage gateway system provided in the second embodiment of the present invention;

[0021] Figure 6 It is a schematic structural diagram of a storage gateway system provided in the third embodiment of the present invention;

[0022] Figure 7 It is another schematic structural diagram of a storage gateway system provided in the third embodiment of the present invention. Detailed implementation manners

[0023] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. It should be understood that the steps recorded in the method embodiments of the present invention can be executed in different orders and / or executed in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this regard.

[0024] As used herein, the term "including" and its variants are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0025] It should be noted that the terms "first", "second", "third", etc. in the specification and claims of the present invention and the above drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variants are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should be noted that the modifiers "one" and "more than one" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] Embodiment 1

[0029] Figure 1 FIG. is a schematic flowchart of a data access method for a storage gateway system provided by Embodiment 1 of the present invention. This method is applicable to the situation where the service end accesses data. This method can be executed by the storage gateway system, where the system can be implemented by software and / or hardware. In this embodiment, the storage gateway can be a NAS gateway.

[0030] As Figure 1 shown, a data access method for a storage gateway system provided by Embodiment 1 of the present invention includes the following steps:

[0031] S110. Perform metadata operations through the file protocol module in the storage gateway system and the cache file subsystem in the storage gateway system.

[0032] Among them, the file protocol module can be used to receive and parse network protocols, and convert network protocols into protocols that can be read by the interface of the cache file subsystem. Network protocols can include SMB protocol, NFS protocol, FTP protocol, etc.

[0033] In this embodiment, the cache file subsystem includes a metadata cache module and a data cache module.

[0034] Among them, by introducing a cache file subsystem in the storage gateway system, when the storage gateway system receives an IO operation request sent by the service end, it can not directly write to the original storage device, but maintain metadata cache and data cache by itself.

[0035] Figure 2 FIG. is a schematic diagram of the metadata cache tree structure in a data access method for a storage gateway system provided by Embodiment 1 of the present invention. As Figure 2 shown, the metadata cache maintains a tree structure in the form of a parent-child link, Figure 2 which is a three-layer tree structure. The top layer is the root node. Each node is composed of a triple, and each node can include the INO of this node, the INO of the parent node, and the name of this node.

[0036] The metadata cache module can be persisted to a high-speed storage device. Exemplarily, the high-speed storage device may include an SSD, NVMESSD, or various NoSQL database storages, such as MongoDB, RocksDB, or Redis. The data cache module can be persisted in a distributed object or file system. In this embodiment, since the metadata cache module is persisted to the high-speed storage device and the data cache module is persisted in the distributed object or file system, the processing speed of the data cache module and the metadata cache module is accelerated, achieving better data access performance.

[0037] In this embodiment, metadata operations may include: after performing protocol conversion through the file protocol module, determining whether the target metadata is cached; if the cache is cached, a metadata request response may be directly returned to the business end; if the cache is not cached, all sub-items of the current level directory may be loaded into the metadata cache module from the back-end massive storage system through the NAS protocol, and the loaded sub-directories may be marked as metadata not cached, and the sub-files may be marked as data not cached, and then the metadata request response may be returned to the business end. At the same time, file logs may be recorded, and the synchronization module in the storage gateway system may sequentially persist metadata modifications to the back-end massive file storage system (Massive FileSystem, MASSFS) according to the file logs. Optionally, the massive file storage system may be a traditional NAS device, a local SAN, an unstructured data management system, or cloud storage.

[0038] S120: Based on the metadata in the metadata cache module after the metadata operation, data access is performed through the file protocol module and the cache file subsystem.

[0039] Among them, data access can include data reading operations and data writing operations. The execution order of data writing operations and data reading operations is not distinguished. Only data writing operations, only data reading operations, or both data writing operations and data reading operations can be executed.

[0040] In this embodiment, when the storage gateway system receives a read request sent by the service end through the file protocol module, it needs to convert the network protocol for sending the read request before performing the data read operation. The data read operation may include the following process: retrieve the corresponding metadata from the metadata cache module after metadata operations according to the INO in the network protocol for sending the read request, and determine whether the data cache is hit according to the metadata. If so, read the data from the data cache module and return a read request response to the service end; if not, read the data from the mass storage system at the back end through the NAS protocol, return a read request response to the service end, and asynchronously load the data from the mass storage system at the back end into the data cache module through the synchronization module in the storage gateway system.

[0041] In this embodiment, when the storage gateway system receives a write request sent by the service end through the file protocol module, it needs to convert the network protocol for sending the write request before performing the data write operation. The data write operation may include the following process: retrieve the corresponding metadata from the metadata cache module after metadata operations according to the INO carried in the network protocol for sending the write request, and determine whether the data cache is hit according to the metadata. If so, directly write the data into the corresponding file in the data cache module; if not, read all the data of the corresponding file from the mass storage system at the back end and write the data into the data cache module, and write the corresponding data update into the corresponding file in the data cache module.

[0042] A data access method for a storage gateway system provided in the first embodiment of the present invention first performs metadata operations through the file protocol module in the storage gateway system and the cache file subsystem in the storage gateway system. The cache file subsystem includes a metadata cache module and a data cache module; then, based on the metadata in the metadata cache module after the metadata operations, data access is performed through the file protocol module and the cache file subsystem. In the case of cache hit, the service IO, that is, the data read operation and the data write operation, only need to go through one network protocol process of the NAS gateway to return. In the case of cache miss, the NAS gateway cache file subsystem will load the metadata from the mass storage system at the back end through the NAS protocol; even in the case of cache miss, it only needs to be loaded once, and subsequent associated IOs will hit in the cache. Using this method can greatly alleviate the IO amplification phenomenon caused by NAS protocol forwarding and improve the access performance of the service system.

[0043] Further, the metadata operation through the file protocol module in the storage gateway system and the cached file subsystem in the storage gateway system includes: after receiving a metadata request sent by the service end through the service system, converting the first network protocol for transmitting the metadata request through the file protocol module in the storage gateway system; obtaining target metadata from the converted first network protocol through the file protocol module; determining whether the target metadata exists in the metadata cache module through the cached file subsystem; if not, loading the target metadata from the backend mass storage system into the metadata cache module through the synchronization module in the storage gateway system, and sending a metadata request response back to the service end through the network.

[0044] Among them, the first network protocol can be understood as the network protocol for transmitting the metadata request, and the first network protocol can be the SMB protocol. The SMB-Server of the file protocol module can receive and parse the SMB protocol, convert it into a protocol that the interface of the cached file subsystem can read, and then the cached file subsystem can perform corresponding operations.

[0045] Among them, the method for determining whether the target metadata exists in the metadata cache module can be: querying whether there is metadata corresponding to the INO in the metadata cache module according to the INO carried in the first network protocol. If so, it is determined that the target metadata exists in the metadata cache module.

[0046] In this embodiment, after obtaining the target metadata, the storage gateway can encapsulate the request response and return it to the service end through the network protocol.

[0047] Further, the method further includes: if so, obtaining the target metadata from the metadata cache module and sending a metadata request response back to the service end through the network.

[0048] Among them, after obtaining the target metadata, the storage gateway can encapsulate the request response and send the encapsulated response request back to the service end through the network.

[0049] In this embodiment, while loading the target metadata from the backend mass storage system into the metadata cache module through the synchronization module in the storage gateway system, a file log is recorded, and the metadata modification in the metadata cache module is persisted to the mass file storage system in the recording order of the file log through the synchronization module.

[0050] In this embodiment, loading the target metadata from the massive storage system at the back end into the metadata cache module through the synchronization module in the storage gateway system includes: loading all subdirectories of the current-level directory from the massive storage system at the back end into the metadata cache module through the synchronization module in the storage gateway system, marking the subdirectories loaded into the metadata cache module as metadata not cached, and marking the sub-files in the sub-targets as data not cached.

[0051] Further, the data access includes a data read operation. Correspondingly, based on the metadata operation, data access through the file protocol module and the cached file subsystem includes: after receiving a read request sent by the service end through the service system, converting the second network protocol for transmitting the read request through the file protocol module in the storage gateway system; retrieving the corresponding first metadata from the metadata cache module after the metadata operation according to the label carried in the converted second network protocol, and querying whether the target read data exists in the data cache module of the cached file subsystem according to the first metadata; if it exists, obtaining the target read data from the data cache module and sending a read request response back to the service end through the network.

[0052] Among them, the second network protocol can be understood as the network protocol for transmitting the read request, and the second network protocol can be the SMB protocol. The first metadata can be understood as the metadata in the metadata cache module with the INO carried in the second network protocol.

[0053] Further, if it does not exist, loading the target read data from the massive storage system into the data cache module and sending a read request response back to the service end through the network.

[0054] In this embodiment, after encapsulating the read request response, the storage gateway system can send the encapsulated read request back to the service end through the network, and at the same time, can update the access times and access time of the target read data.

[0055] Further, the data access includes a data write operation. Correspondingly, based on the metadata operation, data access through the file protocol module and the cached file subsystem includes: after receiving a write request sent by the service end through the service system, converting the third network protocol for transmitting the write request through the file protocol module in the storage gateway system; retrieving the corresponding second metadata from the metadata cache module after the metadata operation according to the label in the converted third network protocol, and querying whether the target write data exists in the data cache module of the cached file subsystem according to the second metadata; if it exists, updating and writing the data corresponding to the target write data into the data cache module.

[0056] Among them, the third network protocol can be understood as the network protocol for transmitting write requests, and the third network protocol can be the SMB protocol. The second metadata can be understood as the metadata in the metadata cache module that has the INO carried in the third network protocol.

[0057] In one embodiment, after writing the corresponding data update into the data cache module, since the target read data is not updated in the global cache, the target read data can be marked as dirty data, and at the same time, the access count and access time of the target read data can be updated.

[0058] Furthermore, if it does not exist, the target write data is written into the data cache module from the mass storage system through the synchronization module, and the data update corresponding to the target write data is written into the data cache module through the synchronization module.

[0059] Among them, the storage gateway can also encapsulate the write request reply and send it back to the service end through the network.

[0060] Furthermore, the least recently used (LRU) strategy can be used to replace the cold data that has not been accessed for a long time from the data cache module, and only the hot data that is frequently accessed is retained. In this way, the cache pressure of the cache module can be effectively relieved.

[0061] Embodiment 2

[0062] Based on the technical solutions of the above embodiments, the embodiments of the present invention provide several specific implementation manners.

[0063] In the following implementation manners, the service end uses a Windows service end, the storage gateway system uses a NAS gateway system, and the backend mass storage system uses the original NAS device.

[0064] Figure 3 It is a flow example diagram of metadata operations in a data access method of a storage gateway system provided by Embodiment 2 of the present invention. As Figure 3 shown, as a specific implementation manner of this embodiment, the metadata operation may include the following steps:

[0065] S401. The Windows service system of the Windows service end sends a Request, that is, a metadata request, to the NAS gateway system through the network using the SMB protocol.

[0066] S402. The SMB-Server of the file protocol module in the NAS gateway system receives and parses the SMB protocol, and converts the SMB protocol into an operation executable by the cache file subsystem.

[0067] S403. The cache file subsystem queries in the MdCache, i.e., the metadata cache module, whether the metadata, i.e., the target metadata, exists.

[0068] If so, execute S404; otherwise, execute S405.

[0069] S404. After obtaining the metadata from the MdCache, the NAS gateway system encapsulates the SMB Reply, i.e., the metadata request reply, and sends it back to the Windows service end through the network, and records the Mdlog, i.e., the file log, corresponding to the metadata modification operation.

[0070] S405. The synchronization module of the NAS gateway system loads the metadata from the original NAS device into the MdCache, and then executes S404.

[0071] Figure 4 This is a flowchart example of the data reading operation in a storage gateway system data access method provided by the second embodiment of the present invention. As Figure 4 shown, as a specific implementation manner of this embodiment, the data reading operation may include the following steps:

[0072] S501. The Windows service system of the Windows service end sends a ReadRequest, i.e., a read request, to the NAS gateway system through the network using the SMB protocol.

[0073] S502. The SMB-Server of the file protocol module in the NAS gateway system receives and parses the SMB protocol, and converts the SMB protocol into an operation executable by the cache file subsystem.

[0074] S503. The cache file subsystem queries in the DataCache, i.e., the data cache module, whether the data, i.e., the target read data, exists.

[0075] If so, execute S504; if not, execute S505.

[0076] S504. Read the data in the DataCache. The NAS gateway system encapsulates the SMB Read Reply, i.e., the read request reply, and sends it back to the Windows service end through the network, and at the same time updates the access times and access time of the data.

[0077] S505. The NAS gateway system reads the corresponding data from the original NAS device, and then the NAS gateway system encapsulates and sends back the fetch request reply to the Windows service end, and at the same time updates the access times and access time of the data.

[0078] Figure 5This is a flowchart example of the data writing operation in the data access method of a storage gateway system provided in the second embodiment of the present invention. As shown in Figure 5 As a specific implementation manner of this embodiment, the data writing operation may include the following steps:

[0079] S601. The Windows business system on the Windows business side sends a WriteRequest, that is, a write request, to the NAS gateway system through the network using the SMB protocol.

[0080] S602. The SMB-Server of the file protocol module in the NAS gateway system receives and parses the SMB protocol, and converts the SMB protocol into an operation executable by the cache file subsystem.

[0081] S603. The cache file subsystem queries in the DataCache whether the data, that is, the target write data, exists.

[0082] If so, execute S604; if not, execute S605.

[0083] S604. Update and write the corresponding data into the corresponding file in the DataCache, mark the data as dirty data, and at the same time update the access count and access time of the data; the NAS gateway system encapsulates the SMB Write Reply, that is, the read request reply, and sends it back to the Windows business side through the network.

[0084] S605. The synchronization module of the NAS gateway system reads all the data of the corresponding file from the original NAS device and writes it into the DataCache, and then executes S604.

[0085] In the data access method of the storage gateway system provided in the second embodiment of the present invention, the synchronization module sequentially executes the metadata operations in the original NAS device according to the Mdlog, so as to persist the dirty data in the MdCache to the original NAS device. At the same time, the synchronization module will obtain the dirty data in the DataCache and use the thread pool to persist all the dirty data to the original NAS device. Since the file may be very large, but the length of the data read and write request may only be a small part of the file, for the case where the read request misses, it will directly read and return from the original NAS device, and then the synchronization module will load the corresponding read data from the original NAS device into the DataCache.

[0086] Embodiment 3

[0087] Figure 6Schematic diagram of a storage gateway system provided in Embodiment 3 of the present invention. This system is applicable to the situation where the business system at the service end accesses data. The system can be implemented by software and / or hardware, and the system can be a NAS gateway system.

[0088] As Figure 6 shown, the system includes a file protocol module 10 and a cached file subsystem 20. The protocol module 10 is connected to the cached file subsystem 20;

[0089] The cached file subsystem 20 includes a synchronization module 23, a metadata cache module 21, and a data cache module 22. The synchronization module 23 is respectively connected to the metadata cache module 21 and the data cache module 22;

[0090] Perform metadata operations through the file protocol module 10 in the storage gateway system and the cached file subsystem 20 in the storage gateway system;

[0091] Based on the metadata in the metadata cache module 21 after the metadata operation, perform data access through the file protocol module 10 and the cached file subsystem 20.

[0092] In this embodiment, the system first performs metadata operations through the file protocol module 10 in the storage gateway system and the cached file subsystem 20 in the storage gateway system; then, based on the metadata in the metadata cache module 21 after the metadata operation, perform data access through the file protocol module 10 and the cached file subsystem 20. Using this system can accelerate the data access speed of the storage gateway, greatly alleviate the IO amplification phenomenon caused by multiple network protocol forwards in the prior art, and improve the access performance of the business system at the service end.

[0093] Further, the performing metadata operations through the file protocol module in the storage gateway system and the cached file subsystem in the storage gateway system includes: after receiving a metadata request sent by the service end through the business system, converting the first network protocol for transmitting the metadata request through the file protocol module 21; obtaining target metadata from the converted first network protocol through the file protocol module 10; determining whether the target metadata exists in the metadata cache module 21 through the cached file subsystem 20; if not, loading the target metadata from the backend mass storage system into the metadata cache module 21 through the synchronization module 23, and sending a metadata request response back to the service end through the network.

[0094] Further, if so, obtain the target metadata from the metadata cache module 21 and send a metadata request response back to the service end through the network.

[0095] Further, while loading the target metadata from the massive storage system at the back end into the metadata cache module 21 through the synchronization module 23, a file log is recorded, and the metadata modification in the metadata cache module 21 is persisted to the massive file storage system by the synchronization module 23 according to the recording order of the file log.

[0096] Further, loading the target metadata from the massive storage system at the back end into the metadata cache module 21 by the synchronization module 23 includes: loading all subdirectories of the current-level directory from the massive storage system at the back end into the metadata cache module 21 by the synchronization module 23, marking the loaded subdirectories in the metadata cache module 21 as metadata not cached, and marking the sub-files in the sub-targets as data not cached.

[0097] Further, the data access includes data read operations. Correspondingly, based on the metadata operations, data access through the file protocol module 10 and the cache file subsystem 20 includes:

[0098] After receiving a read request sent by the service end through the service system, converting the second network protocol for transmitting the read request through the file protocol module 10;

[0099] Retrieving corresponding first metadata from the metadata cache module 21 after the metadata operation according to the mark carried in the converted second network protocol, and querying whether the target read data exists in the data cache module 22 of the cache file subsystem 20 according to the first metadata;

[0100] If it exists, obtaining the target read data from the data cache module 22 and sending a read request response back to the service end through the network.

[0101] If it does not exist, loading the target read data from the massive storage system into the data cache module 22 and sending a read request response back to the service end through the network.

[0102] Further, the data access includes data write operations. Correspondingly, based on the metadata operations, data access through the file protocol module 10 and the cache file subsystem 20 includes:

[0103] After receiving a write request sent by the service end through the service system, converting the third network protocol for transmitting the write request through the file protocol module 10 in the storage gateway system;

[0104] Retrieve the corresponding second metadata from the metadata cache module 21 after the metadata operation according to the tags in the converted third network protocol, and query whether there is target write data in the data cache module 22 of the cache file subsystem 20 according to the second metadata;

[0105] If it exists, update and write the data corresponding to the target write data into the data cache module 22;

[0106] If it does not exist, cache the target write data into the data cache module 22 from the mass storage system through the synchronization module 23, and update and write the data corresponding to the target write data into the data cache module 22 through the synchronization module 23.

[0107] It should be noted that the metadata cache module 22 can be persisted to a high-speed storage device, and the data cache module 22 can be persisted in a distributed object or file system.

[0108] It should be noted that the data cache module 22 can also be used to replace cold data that has not been accessed for a long time from the data cache module using the least recently used policy, and only retain hot data that is frequently accessed.

[0109] The above storage gateway system can execute the data access method of the storage gateway system provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0110] Figure 7 Another structural schematic diagram of a storage gateway system provided in Embodiment 3 of the present invention is shown in Figure 7 As shown, the metadata cache module in this system is persisted in MongoDB on the SSD, and the data cache module is persisted in the distributed object storage Rados built on the SSD. Through this system, the data access method of the storage gateway system provided in Embodiment 1 and Embodiment 2 can be executed. This method performs metadata operations, data reading operations, and data writing operations through this system. The specific execution process will not be elaborated here, and reference can be made to the content of Embodiment 1 and Embodiment 2.

[0111] It should be understood that various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in the present invention can be executed in parallel, sequentially, or in a different order, as long as the results expected by the technical solution of the present invention can be achieved. There is no limitation here.

[0112] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data access method for a storage gateway system, characterized in that Applied to a storage gateway system, the method includes: Perform metadata operations through a file protocol module in the storage gateway system and a cache file subsystem in the storage gateway system. The cache file subsystem includes a synchronization module, a metadata cache module, and a data cache module; Based on the metadata in the metadata cache module after the metadata operation, perform data access through the file protocol module and the cache file subsystem; The data access includes a data write operation. Correspondingly, the performing data access through the file protocol module and the cache file subsystem based on the metadata in the metadata cache module after the metadata operation includes: After receiving a write request sent by a service end through a service system, convert a third network protocol for transmitting the write request through the file protocol module in the storage gateway system; the third network protocol is the network protocol for transmitting the write request; Retrieve corresponding second metadata from the metadata cache module after the metadata operation according to a tag in the converted third network protocol, and query whether target write data exists in the data cache module of the cache file subsystem according to the second metadata; the second metadata is the metadata in the metadata cache module with an INO carried in the third network protocol; If it exists, update and write the data corresponding to the target write data into the data cache module; If it does not exist, write the target write data from the backend mass storage system into the data cache module through the synchronization module, and update and write the data corresponding to the target write data into the data cache module through the synchronization module.

2. The method according to claim 1, wherein The performing metadata operations through the file protocol module in the storage gateway system and the cache file subsystem in the storage gateway system includes: After receiving a metadata request sent by a service end through a service system, convert a first network protocol for transmitting the metadata request through the file protocol module in the storage gateway system; the first network protocol is the network protocol for transmitting the metadata request; Obtain target metadata from the converted first network protocol through the file protocol module; Determine whether the target metadata exists in the metadata cache module through the cache file subsystem; If not, load the target metadata from the backend mass storage system into the metadata cache module through the synchronization module in the storage gateway system, and send a metadata request response back to the service end through the network.

3. The method according to claim 2, wherein The method further includes: If so, obtain the target metadata from the metadata cache module, and send a metadata request response back to the service end through the network.

4. The method according to claim 2, characterized in that, While loading the target metadata from the backend mass storage system into the metadata cache module through the synchronization module in the storage gateway system, record a file log, and persistently store the metadata modification in the metadata cache module into the mass storage system in the recording order of the file log through the synchronization module.

5. The method according to claim 2, wherein Loading the target metadata from the massive storage system at the back end into the metadata cache module through the synchronization module in the storage gateway system includes: Loading all subdirectories of the current-level directory from the massive storage system at the back end into the metadata cache module through the synchronization module in the storage gateway system, marking the subdirectories loaded into the metadata cache module as metadata not cached, and marking the sub-files in the sub-targets as data not cached.

6. The method according to claim 1, characterized in that The data access includes data read operations. Correspondingly, the data access based on the metadata in the metadata cache module after the metadata operation is performed through the file protocol module and the cached file subsystem, and includes: After receiving a read request sent by the service end through the service system, converting the second network protocol for transmitting the read request through the file protocol module in the storage gateway system; the second network protocol is the network protocol for transmitting the read request. Retrieving the corresponding first metadata from the metadata cache module after the metadata operation according to the tag carried in the converted second network protocol, and querying whether the target read data exists in the data cache module of the cached file subsystem according to the first metadata; the first metadata is the metadata in the metadata cache module with the INO carried in the second network protocol. If it exists, obtaining the target read data from the data cache module and sending a read request response back to the service end through the network.

7. The method according to claim 6, wherein The method further includes: If it does not exist, loading the target read data from the massive storage system into the data cache module and sending a read request response back to the service end through the network.

8. A storage gateway system, characterized in that, The system includes a file protocol module and a cached file subsystem, and the protocol module is connected to the cached file subsystem. The cached file subsystem includes a synchronization module, a metadata cache module, and a data cache module, and the synchronization module is respectively connected to the metadata cache module and the data cache module. Performing metadata operations through the file protocol module in the storage gateway system and the cached file subsystem in the storage gateway system. Based on the metadata in the metadata cache module after the metadata operation, performing data access through the file protocol module and the cached file subsystem. The data access includes data write operations. Correspondingly, the data access based on the metadata in the metadata cache module after the metadata operation is performed through the file protocol module and the cached file subsystem, and includes: After receiving a write request sent by the service end through the service system, converting the third network protocol for transmitting the write request through the file protocol module in the storage gateway system. Retrieve the corresponding second metadata from the metadata cache module after the metadata operation according to the tag in the converted third network protocol, and query whether there is target write data in the data cache module of the cache file subsystem according to the second metadata; the third network protocol is the network protocol for transmitting write requests; the second metadata is the metadata in the metadata cache module with the INO carried in the third network protocol; If it exists, update and write the data corresponding to the target write data into the data cache module; If it does not exist, write the target write data into the data cache module from the backend mass storage system through the synchronization module, and update and write the data corresponding to the target write data into the data cache module through the synchronization module.

Images