Method, apparatus, gateway, medium and product for global storage data consistency

Abstract

The present invention discloses a method, apparatus, gateway, medium and product for global storage data consistency. The method includes: obtaining a preset number of operation logs corresponding to metadata operations; adding the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronizing the metadata operations corresponding to the file creation operation logs from the preset queue to a global memory; synchronizing the metadata operations corresponding to the other operation logs except the file creation operation logs in the preset number of operation logs to the global memory in sequence. By adding the file creation operation logs to the preset queue, this method can improve the efficiency of caching metadata updates to the global memory.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of gateway data storage, and in particular, to a method, apparatus, gateway, medium and product for global storage data consistency. Background Art

[0002] With the development of information technology, the unstructured data of enterprises has grown rapidly. With the growth of data, the pressure on the backend storage will become greater and greater, and the performance of unstructured storage devices under massive data will decline significantly. Therefore, it is necessary to provide a cache through a local storage gateway to accelerate the storage access performance. Therefore, the data consistency of updating the cache data to the global storage must be ensured.

[0003] In order to ensure the data consistency of updating the cache data to the global storage, the prior art generally records file operation logs in the storage gateway, and then sequentially updates the metadata operations in the cache to the global storage according to the file operation logs.

[0004] However, the speed of updating the metadata operations in the cache to the global storage in the prior art is relatively slow. Summary of the Invention

[0005] The present invention provides a method, apparatus, gateway, medium and product for global storage data consistency to solve the problem that the speed of updating the metadata operations in the cache of the storage gateway to the global storage in the prior art is relatively slow.

[0006] According to an aspect of the present invention, a method for global storage data consistency is provided, including:

[0007] Obtaining a preset number of operation logs, where the operation logs correspond to metadata operations;

[0008] Adding the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronizing the metadata operations corresponding to the file creation operation logs from the preset queue to the global memory;

[0009] Synchronizing the metadata operations corresponding to the other operation logs except the file creation operation logs in the preset number of operation logs to the global memory in sequence.

[0010] According to another aspect of the present invention, an apparatus for global storage data consistency is provided, including:

[0011] An obtaining module, configured to obtain a preset number of operation logs, where the operation logs correspond to metadata operations;

[0012] The first synchronization module is configured to add the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronize the metadata operations corresponding to the file creation operation logs from the preset queue to the global memory;

[0013] The second synchronization module is configured to synchronize, in sequence, the metadata operations corresponding to the other operation logs except the file creation operation logs in the preset number of operation logs to the global memory.

[0014] According to another aspect of the present invention, there is provided a storage gateway, which includes: at least one processor; and

[0015] a memory communicatively connected to the at least one processor; wherein,

[0016] the memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, the at least one processor is enabled to execute the method for global storage data consistency according to any embodiment of the present invention.

[0017] According to another aspect of the present invention, there is provided a computer-readable storage medium storing computer instructions for enabling a processor to implement the method for global storage data consistency according to any embodiment of the present invention when executed.

[0018] The technical solution of the embodiment of the present invention solves the problem of slow speed in updating the metadata operations in the storage gateway cache to the global storage by adding the file creation operation logs to the preset queue, and achieves the beneficial effect of improving the efficiency of updating the cached metadata to the global memory.

[0019] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present invention, nor is it used to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0021] Figure 1 It is a flowchart showing a method for global storage data consistency provided by Embodiment 1 of the present invention;

[0022] Figure 2 It is a schematic flowchart of a method for global storage data consistency provided in the second embodiment of the present invention;

[0023] Figure 3 It is a schematic flowchart of a method for global storage data consistency provided in the exemplary embodiment of the present invention;

[0024] Figure 4 It is a schematic flowchart of synchronizing metadata operations corresponding to operation logs to a global memory provided in the exemplary embodiment of the present invention;

[0025] Figure 5 It is a schematic structural diagram of a device for global storage data consistency provided in the third embodiment of the present invention;

[0026] Figure 6 It is a schematic structural diagram of a storage gateway for a method of global storage data consistency according to an embodiment of the present invention. Detailed implementation manners

[0027] In order to enable those skilled in the art to better understand the solution of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention. It should be understood that the steps recorded in the method embodiments of the present invention can be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this regard.

[0028] The term "including" and its variants used herein are open-ended, that is, "including but not limited to". The term "based on" is "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0029] It should be noted that the terms "first", "second", etc. in the description, claims and above-mentioned drawings of the present invention are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device comprising a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless clearly specified otherwise in the context, it should be understood as "one or more".

[0031] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0032] Embodiment 1

[0033] Figure 1 FIG. is a schematic flowchart of a method for global storage data consistency provided by Embodiment 1 of the present invention. This method is applicable to the situation where cache metadata in a storage gateway is updated to a global memory for global storage. This method can be executed by a device for global storage data consistency, where the device can be implemented by software and / or hardware and is generally integrated in a storage gateway. In this embodiment, the storage gateway includes but is not limited to a NAS gateway.

[0034] As Figure 1 shown, a method for global storage data consistency provided by Embodiment 1 of the present invention includes the following steps:

[0035] S110. Obtain a preset number of operation logs.

[0036] Among them, the operation logs correspond to metadata operations. The number of obtained operation logs can be set by itself, and the preset number can be any integer greater than 1.

[0037] In this embodiment, there is no specific limitation on the way to obtain operation logs. Multiple operation logs can be obtained through any feasible way. Exemplarily, a feasible way is to pull multiple operation logs from the metadata cache.

[0038] Among them, the metadata cache can be persisted to various NoSQL databases, such as MongoDB, RocksDB, Redis, etc. Among them, the persistence of operation logs can adopt a message queue, a database, or other components that can ensure order.

[0039] S120. Add the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronize the metadata operations corresponding to the file creation operation logs from the preset queue to the global memory.

[0040] Among them, the preset number of operation logs can include various operation logs. Since the metadata cache is a tree structure, operation logs with less impact on the tree structure can be added to the preset queue, and the file creation operation logs can be used as operation logs with less impact on the tree structure.

[0041] In this embodiment, before synchronizing the metadata operations corresponding to the preset number of operation logs to the global memory, a queue can be created in advance, operation logs with less impact on the tree structure are added to the queue, and then the operation logs with less impact on the tree structure are obtained from the queue. According to the order of log acquisition, the metadata operations corresponding to the operation logs with less impact on the tree structure are synchronized to the global memory.

[0042] Among them, the global memory can globally store cached metadata, and the global memory can be any one of a storage device, an unstructured data management system, and cloud storage.

[0043] Further, synchronizing the metadata operations corresponding to the file creation operation logs from the preset queue to the global memory includes: using multiple worker threads in a thread pool to concurrently synchronize multiple metadata operations corresponding to multiple file creation operation logs in the preset queue to the global memory.

[0044] In this embodiment, by creating a thread pool, multiple metadata operations corresponding to multiple file creation operation logs can be concurrently synchronized to the global memory for global storage.

[0045] In one embodiment, the number of worker threads in the thread pool can be dynamically configured. All worker threads can obtain file creation operation logs from the preset queue and perform corresponding data update operations according to the file creation operation logs. All worker threads in the thread pool can concurrently update the metadata operations corresponding to multiple file creation operation logs to the global memory. Among them, when obtaining file creation operation logs from the preset queue, the preset queue can be locked to prevent obtaining duplicate file creation operation logs.

[0046] In this embodiment, synchronizing the metadata operation corresponding to the file creation operation log to the global memory may include: updating the metadata cache in the global memory according to the complete cache path corresponding to the metadata operation that meets the preset conditions and the metadata operation; adding the operation log corresponding to the metadata operation that does not meet the preset conditions to a preset data structure.

[0047] Among them, the preset conditions include: the nodes of the metadata operation do not include the nodes existing in the preset data structure, and the parent paths of the metadata operation do not exist in the preset data structure. Exemplarily, when the nodes of the metadata operation include three nodes A, B, and C, if one of the three nodes A, B, and C exists in the preset data structure, it indicates that the metadata operation does not meet the preset conditions, and the operation log corresponding to the metadata operation needs to be added to the preset data structure; if the three nodes A, B, and C do not exist in the preset data structure, continue to find the parent path upward in the metadata cache. If the parent paths of the metadata operation do not exist in the preset data structure, it indicates that the metadata operation meets the preset conditions, and a complete cache path can be spliced. The metadata in the global memory is updated through the spliced complete cache path and the metadata operation.

[0048] S130. Synchronize the metadata operations corresponding to the other operation logs except the file creation operation log among the preset number of operation logs to the global memory in sequence for each of the other operation logs.

[0049] Among them, the metadata operations corresponding to the other operation logs may include operations that have a greater impact on the tree structure, such as directory Create, Rename, and Remove.

[0050] In this embodiment, the metadata operations corresponding to the other operation logs can be synchronized to the global memory in the arrangement order of the operation logs.

[0051] It can be understood that the process of synchronizing the metadata operations corresponding to other operation logs to the global memory is the same as the process of synchronizing the metadata operations corresponding to the file creation operation logs in step S120, which may specifically include the following process: Determine whether the nodes in the metadata operation include nodes existing in the preset data structure. If so, add the operation log corresponding to the metadata operation to the preset data structure; if not, search for the parent path upward in the metadata cache and determine whether there is a parent path existing in the preset data structure. If so, add the operation log corresponding to the metadata operation to the preset data structure; if not, determine whether a complete cache path can be spliced. If a complete cache path can be spliced, update the metadata in the global memory through the spliced complete cache path and the metadata operation; if a complete cache path cannot be spliced, continue to search for the parent path upward in the metadata cache until a complete cache path is spliced.

[0052] It should be noted that the execution of S120 and S130 does not distinguish the order and can also be executed simultaneously.

[0053] A method for global storage data consistency provided in Embodiment 1 of the present invention first obtains a preset number of operation logs; then adds the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronizes the metadata operations corresponding to the file creation operation logs from the preset queue to the global memory; finally, synchronizes the metadata operations corresponding to the other operation logs except the file creation operation logs in the preset number of operation logs to the global memory in sequence. This method can improve the efficiency of updating cached metadata to the global memory by adding the file creation operation logs to the preset queue.

[0054] Embodiment 2

[0055] Figure 2 FIG. is a schematic flowchart of a method for global storage data consistency provided in Embodiment 2 of the present invention. Embodiment 2 is optimized based on the above embodiments. In this embodiment, a preset data structure is further introduced. For the content not detailed in this embodiment, please refer to Embodiment 1.

[0056] As Figure 2 shown, a method for global storage data consistency provided in Embodiment 2 of the present invention includes the following steps:

[0057] S210. Obtain a preset number of operation logs.

[0058] S220. Add the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronize the metadata operations corresponding to the file creation operation logs from the preset queue to the global memory.

[0059] S230. Synchronize the metadata operations corresponding to each of the other operation logs among the preset number of operation logs, except for the file creation operation logs, to the global memory in sequence.

[0060] S240. Add the operation logs that failed to be synchronized among the preset number of operation logs to a preset data structure. After processing the preset number of operation logs, synchronize the metadata operations corresponding to the operation logs that failed to be synchronized in the preset data structure to the global memory, and delete the operation logs corresponding to the successfully synchronized metadata operations from the preset data structure.

[0061] In one embodiment, if an operation log fails for some reason during processing, the operation log can be added to a preset data structure. Among them, the preset data structure can be a preset data structure with a search function.

[0062] Among them, when processing other operation logs subsequently, if it is found that the parent path of the metadata operation corresponding to the operation log is in the preset data structure, the operation log can be added to the preset data structure.

[0063] It should be noted that adding to the preset data structure can ensure that when an operation log fails to execute, it only affects the update of other data in its subtree, while the bypass data can still be updated to the global memory.

[0064] A method for global storage data consistency provided in the second embodiment of the present invention. On the one hand, the method adds the file creation operation logs to a preset queue, and by introducing a thread pool, the metadata operations corresponding to the file creation operation logs can be executed concurrently, which can improve the efficiency of caching metadata updates to the global memory; on the other hand, by introducing a preset data structure, when the metadata operation corresponding to an operation log fails to be synchronized, the subtree with synchronization restrictions can be skipped, so that the bypass data is not affected.

[0065] Based on the technical solutions of the above embodiments, the embodiments of the present invention provide a specific implementation manner.

[0066] As a specific implementation manner of this embodiment, Figure 3 is a flowchart of a method for global storage data consistency provided by the exemplary embodiment of the present invention. As Figure 3 shown, the method may include the following steps:

[0067] S10-1. Pull N operation logs at one time.

[0068] S10-2. Pull out the first unprocessed operation log among the N operation logs in sequence.

[0069] S103. Determine whether this operation log is a File Create.

[0070] If it is, execute S107; if not, execute S104.

[0071] S104. Update the metadata operation corresponding to this operation log to the global memory.

[0072] S105. Determine whether all the N logs fetched have been processed.

[0073] If it is, execute S102; if not, execute S106.

[0074] S106. Process the operation logs in the DelayMap, that is, in the preset data structure, delete the successfully processed operation logs from the DelayMap, and keep the operation logs that still fail to be processed in the DelayMap.

[0075] S107. Add this operation log to a queue.

[0076] S108. Fetch operation logs from the queue, start another thread pool in the program, the number of workers in the thread pool can be dynamically configured, and synchronize the metadata operations corresponding to the operation logs to the global memory through multiple workers.

[0077] Furthermore, Figure 4 This is a schematic flow diagram of synchronizing the metadata operation corresponding to the operation log to the global memory provided by the exemplary embodiment of the present invention. The steps of synchronizing the metadata operation corresponding to the operation log to the global storage are as Figure 4 shown, and the specific steps are as follows:

[0078] S201. Check whether there is an entry in the DelayMap that affects this node.

[0079] If it is, execute S207; if not, execute S202.

[0080] S202. Search upward for the parent path in the MdCache, that is, the metadata cache.

[0081] S203. Check whether there is an entry in the DelayMap that affects the parent path.

[0082] If there is, execute S207; if not, execute S204.

[0083] S204. Determine whether the path has been completely concatenated.

[0084] If it is, execute S205; if not, return and execute S202 - S204 again.

[0085] S205. Update the metadata in the global memory through the assembled complete path and the corresponding metadata operations.

[0086] S206. Determine whether the update operation is successful.

[0087] If successful, end; if failed, execute S207.

[0088] S207. Add this operation log to the DelayMap.

[0089] The above method inserts the File Create operation into the thread pool for concurrent execution, improving the speed of cache metadata update. And through the decomposition of the tree structure, if a certain metadata update fails, all metadata update operations of its subtree are skipped, so that the update exception does not block the overall data update, improving the efficiency of updating cache data to the global storage.

[0090] Embodiment III

[0091] Figure 5 FIG. is a schematic structural diagram of an apparatus for global storage data consistency provided in Embodiment III of the present invention. This apparatus is applicable to the situation of updating the cache metadata in the storage gateway to the global memory for global storage, where the apparatus can be implemented by software and / or hardware and is generally integrated on the storage gateway.

[0092] As Figure 5 shown, the apparatus includes: an acquisition module 110, a first synchronization module 120, and a second synchronization module 130.

[0093] The acquisition module 110 is configured to acquire a preset number of operation logs, and the operation logs correspond to metadata operations;

[0094] The first synchronization module 120 is configured to add the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronize the metadata operations corresponding to the file creation operation logs from the preset queue to the global memory;

[0095] The second synchronization module 130 is configured to synchronize the other operation logs except the file creation operation logs in the preset number of operation logs to the global memory in sequence according to the order of the other operation logs corresponding to the metadata operations.

[0096] In this embodiment, the device first obtains a preset number of operation logs through the obtaining module 110; then the first synchronization module 120 adds the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronizes the metadata operations corresponding to the file creation operation logs from the preset queue to the global memory; finally, the second synchronization module 130 synchronizes the other operation logs in the preset number of operation logs except the file creation operation logs, and synchronizes the metadata operations corresponding to the other operation logs to the global memory in sequence.

[0097] This embodiment provides a device for global storage data consistency, which can improve the efficiency of caching metadata updates to the global memory.

[0098] Furthermore, the global memory includes one of a storage device, an unstructured data management system, and cloud storage.

[0099] Furthermore, the first synchronization module 120 is specifically configured to: through multiple worker threads in the thread pool, synchronize multiple metadata operations corresponding to multiple file creation operation logs in the preset queue to the global memory in a concurrent manner.

[0100] Furthermore, the device further includes a processing module, configured to add the operation logs that fail to be synchronized in the preset number of operation logs to a preset data structure. After the preset number of operation logs are processed, synchronize the metadata operations corresponding to the operation logs that fail to be synchronized in the preset data structure to the global memory, and delete the operation logs corresponding to the metadata operations that are successfully synchronized from the preset data structure.

[0101] Furthermore, synchronizing the metadata operation corresponding to the operation log to the global memory includes: if the metadata operation corresponding to the operation log meets a preset condition, determining the complete cache path corresponding to the metadata operation that meets the preset condition; updating the cache in the global memory according to the complete cache path and the metadata operation; where the preset condition includes: the nodes in the metadata operation do not include the nodes existing in the preset data structure, and the parent paths of the metadata operation do not exist in the preset data structure.

[0102] Furthermore, the device adds the operation logs corresponding to the metadata operations that do not meet the preset condition to the preset data structure.

[0103] The above device for global storage data consistency can execute the method for global storage data consistency provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0104] Embodiment 4

[0105] Figure 6 The structural schematic diagram of the storage gateway 10 that can be used to implement the embodiments of the present invention is shown. As Figure 6 shown, the storage gateway 10 includes at least one processor 11 and a memory communicatively connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc. Among them, the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the storage gateway 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0106] Multiple components in the storage gateway 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disc, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0107] The processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 11 include but are not limited to a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for global storage data consistency.

[0108] In some embodiments, a method for global storage data consistency can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the storage gateway 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps in the method for global storage data consistency described above can be executed. Alternatively, in other embodiments, the processor 11 can be configured to execute a method for global storage data consistency by any other appropriate means (for example, by means of firmware).

[0109] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.

[0110] The computer programs for implementing the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer programs, when executed by the processor, cause the functions / operations specified in the flowchart and / or block diagram to be implemented. The computer programs can be executed entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

[0111] In the context of the present invention, a computer-readable storage medium can be a tangible medium that can contain, or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable storage medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) through which the user can provide input to the electronic device. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0113] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which the user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), blockchain network, and the Internet.

[0114] A computing system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system and solves the defects of difficult management and weak business scalability existing in traditional physical hosts and VPS services.

[0115] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is made herein.

[0116] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. A method for global storage data consistency, characterized in that The method includes: Obtaining a preset number of operation logs, where the operation logs correspond to metadata operations; Adding the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronizing the multiple metadata operations corresponding to the multiple file creation operation logs in the preset queue to the global memory concurrently through multiple worker threads in a thread pool; wherein, the file creation operation log is an operation log that has less impact on the tree structure; Synchronizing the metadata operations corresponding to the other operation logs except the file creation operation logs in the preset number of operation logs to the global memory in sequence; Adding the operation logs that fail to be synchronized in the preset number of operation logs to a preset data structure, and when the processing of the preset number of operation logs is completed, synchronizing the metadata operations corresponding to the operation logs that fail to be synchronized in the preset data structure to the global memory, and deleting the operation logs corresponding to the successfully synchronized metadata operations from the preset data structure.

2. The method according to claim 1, characterized in that, The global memory includes one of a storage device, an unstructured data management system, and cloud storage.

3. The method according to claim 1, characterized in that Synchronizing the metadata operation corresponding to the operation log to the global memory includes: If the metadata operation corresponding to the operation log meets a preset condition, determining the complete cache path corresponding to the metadata operation that meets the preset condition; Updating the cache in the global memory according to the complete cache path and the metadata operation; Wherein, the preset condition includes: the nodes in the metadata operation do not include the nodes existing in the preset data structure, and the parent paths of the metadata operation do not exist in the preset data structure.

4. The method according to claim 3, characterized in that It further includes: Adding the operation logs corresponding to the metadata operations that do not meet the preset condition to the preset data structure.

5. A device for global storage data consistency, characterized in that, The apparatus includes: An obtaining module, configured to obtain a preset number of operation logs, where the operation logs correspond to metadata operations; A first synchronization module, configured to add the file creation operation logs included in the preset number of operation logs to a preset queue, and synchronize the multiple metadata operations corresponding to the multiple file creation operation logs in the preset queue to the global memory concurrently through multiple worker threads in a thread pool; wherein, the file creation operation log is an operation log that has less impact on the tree structure; A second synchronization module, configured to synchronize the metadata operations corresponding to the other operation logs except the file creation operation logs in the preset number of operation logs to the global memory in sequence; A processing module, configured to add the operation logs that fail to be synchronized in the preset number of operation logs to a preset data structure, and when the processing of the preset number of operation logs is completed, synchronizing the metadata operations corresponding to the operation logs that fail to be synchronized in the preset data structure to the global memory, and deleting the operation logs corresponding to the successfully synchronized metadata operations from the preset data structure.

6. A storage gateway, characterized in that The storage gateway includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and when the computer program is executed by the at least one processor, enables the at least one processor to execute the method for global storage data consistency according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for implementing the method for global storage data consistency according to any one of claims 1-4 when the computer instructions are executed by a processor.

8. A computer program product, characterized in that, The computer program product includes a computer program which, when executed by a processor, implements the method for global storage data consistency according to any one of claims 1-4.

Images