Data synchronization method, electronic device, and storage medium

Abstract

A data synchronization method, an electronic device, and a storage medium. The data synchronization method comprises: determining an operating state of a master node and a slave node that provide a gateway service for a storage system; when the operating state is switched from a master node standalone operating state to a master-slave replication operating state, screening out target data from local cache data of the master node, wherein the target data is not successfully uploaded to the storage system and is not successfully cached to the slave node; and starting a data cache synchronization task, so as to synchronously cache the target data from the master node to the slave node.

Description

Data synchronization method, electronic device and storage medium

[0001] The present disclosure claims priority to Chinese Patent Application No. 202410969598.9, filed on July 18, 2024 with the Chinese Patent Office, entitled "Data synchronization method, electronic device and storage medium", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] One or more embodiments of the present disclosure relate to the technical field of data synchronization, and in particular to a data synchronization method, an electronic device and a storage medium. BACKGROUND

[0003] A gateway service can bridge between a user-side device and a storage system, facilitating the user-side device to access the storage system. For example, in a cloud storage scenario, the gateway service can provide an interface similar to a traditional file system or block storage, so that the user-side device can access the storage system in a local storage manner. The gateway service can also provide a caching function to reduce frequent access of the user-side device to the storage system, to speed up the read and write speed of data.

[0004] The gateway service specifically provides related functions by a corresponding deployed gateway instance. Each gateway instance can be referred to as a node. In a traditional technical solution, the gateway service only contains a single node. If the single node fails, the dirty data obtained from the user-side device and cached locally by the single node, which has not been uploaded to the storage system, will have a risk of data loss. At this time, even if the user redeploys a new node and binds it to the same storage system, the dirty data in the failed node cannot be accessed, resulting in data inconsistency.

[0005] To this end, the related technology proposes a gateway service using a master-slave node architecture, which provides a gateway service to the storage system by a master node and a slave node, including locally caching data provided by the user-side device, and needs to keep the data on the master node and the slave node completely consistent. Then, after the slave node first accesses or resumes access to the master node, full data synchronization needs to be performed between the master node and the slave node, resulting in too high bandwidth occupation of the data synchronization process and too long time consumption of the entire process. SUMMARY

[0006] To solve the technical problem that, in the related technology, after the slave node first accesses or resumes access to the master node, full data synchronization needs to be performed between the master node and the slave node, resulting in too high bandwidth occupation of the data synchronization process and too long time consumption of the entire process, one or more embodiments of the present disclosure provide a data synchronization method, an electronic device and a storage medium.

[0007] To achieve the above object, one or more embodiments of the present disclosure provide technical solutions as follows.

[0008] According to a first aspect of one or more embodiments of the present disclosure, a data synchronization method is provided, comprising:

[0009] determining a running state of master-slave nodes providing gateway services for a storage system;

[0010] in a case where the running state is switched from a master node independent running state to a master-slave replication running state, screening target data from local cache data of the master node, wherein the target data is neither successfully uploaded to the storage system nor successfully cached to a slave node;

[0011] starting a data cache synchronization task to synchronize the target data from the master node to the slave node.

[0012] According to a second aspect of one or more embodiments of the present disclosure, a data synchronization method of a cloud storage gateway is provided, the cloud storage gateway comprising master nodes and slave nodes for providing cloud storage gateway services to an object storage system; the method comprising:

[0013] determining a running state of master-slave nodes providing cloud storage gateway services for an object storage system;

[0014] in a case where the running state is switched from a master node independent running state to a master-slave replication running state, screening target data from local cache data of the master node, wherein the target data is neither successfully uploaded to the storage system nor successfully cached to a slave node;

[0015] starting a data cache synchronization task to synchronize the target data from the master node to the slave node.

[0016] According to a third aspect of one or more embodiments of the present disclosure, an electronic device is provided, comprising:

[0017] a processor;

[0018] a memory for storing processor executable instructions;

[0019] wherein the processor implements the method of the first aspect or the second aspect by running the executable instructions.

[0020] According to a fourth aspect of one or more embodiments of the present disclosure, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method of the first aspect or the second aspect.

[0021] According to a fifth aspect of one or more embodiments of the present disclosure, a computer readable storage medium is provided, which stores computer instructions, and the computer instructions are executed by a processor to implement the method according to the first aspect or the second aspect.

[0022] It can be seen that, in the technical solution of the present disclosure, the technical prejudice for data consistency in the related art is eliminated, and a brand-new understanding of data consistency is proposed: for a certain data, as long as the data is stored in the majority (i.e., at least two of the master node, the slave node, and the storage system) among the master node, the slave node, and the storage system, the high availability and data consistency for the data can be ensured. Therefore, when the running state between the master and slave nodes is switched from the master independent running state to the master-slave replication running state, only the target data that is neither successfully uploaded to the storage system nor successfully cached to the slave node needs to be determined from the master node, and the target data is cached to the slave node, so that the requirement for data consistency in the present disclosure can be met. For other data, such as data that is stored in the master node and the storage system although not on the slave node, synchronization to the slave node is not required, so that the amount of data synchronized from the master node to the slave node can be effectively reduced, and the bandwidth occupancy rate of data synchronization can be reduced and the time consumption of data synchronization can be reduced while ensuring the high availability of the storage service and preventing the target data from being lost. BRIEF DESCRIPTION OF DRAWINGS

[0023] FIG. 1 is a schematic diagram of an architecture of a data storage system according to an example embodiment.

[0024] FIG. 2 is a flowchart of a data synchronization method according to an example embodiment.

[0025] FIG. 3 is a schematic diagram of data writing in a master-slave node running state according to an example embodiment.

[0026] FIG. 4 is a schematic diagram of Chunk information in different states according to an example embodiment.

[0027] FIG. 5 is a schematic diagram of data reading from a master node by a user according to an example embodiment.

[0028] FIG. 6 is a flowchart of a data synchronization method of a cloud storage gateway according to an example embodiment.

[0029] FIG. 7 is a flowchart of a data synchronization method of a cloud storage gateway according to an example embodiment.

[0030] FIG. 8 is a schematic diagram of a structure of a device according to an example embodiment.

[0031] FIG. 9 is a block diagram of a data synchronization apparatus according to an example embodiment.

[0032] FIG. 10 is a block diagram of a data synchronization apparatus of a cloud storage gateway according to an example embodiment. DETAILED DESCRIPTION

[0033] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, execution and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation portals for the user to choose to authorize or refuse.

[0034] FIG. 1 is an architecture diagram of a data storage system according to an example embodiment. As shown in FIG. 1, the data storage system can include a gateway service 11 and a storage system 12, and the gateway service 11 can include a node 111 and a node 112.

[0035] The storage system 12 can be any system for providing storage services, such as an enterprise-level or network storage system. For example, in a cloud storage scenario, the storage system 12 can be a cloud storage system, such as an OSS (Object Storage Service).

[0036] In cooperation with the storage system 12, the gateway service 11 can bridge between the user-side device and the storage system 12, facilitating the user-side device to access the storage system 12. For example, in a cloud storage scenario, the gateway service 11 can provide an interface similar to a traditional file system or block storage, so that the user-side device can access the storage system 12 in a local storage manner. The gateway service 11 can also provide a caching function to reduce the frequent access of the user-side device to the storage system, to speed up the read and write speed of the data. The gateway service 11 specifically provides the related functions by the corresponding deployed gateway instances. Each gateway instance can be referred to as a node, such as the node 111 and the node 112 shown in FIG. 1.

[0037] Assuming that the node 111 is the master node and the node 112 is the slave node, in the case that both the node 111 and the node 112 are in normal operation, it can be referred to as a Master-Slave Replication operation state. In the Master-Slave Replication operation state, the master node usually responds to the data read / write requests issued by the user. Taking the write process as an example, the node 111 can obtain the write request from the user, store the specified data corresponding to the write request to the local cache of the node 111, and then store it to the node 112 and the storage system 12 respectively. In addition, the data obtained by the master node is replicated to the slave node in real time or close to real time, so as to realize data redundancy. Moreover, when the master node fails, the slave node can replace the master node to respond to the data read / write requests issued by the user, so as to realize failover. It can be seen that in the Master-Slave Replication operation state, the high availability requirement of the database can be met to a certain extent.

[0038] If the slave node fails or cannot continue to provide gateway services due to other reasons, and only the master node provides gateway services alone, it can be referred to as a master node independent operation state. Assuming that the node 111 is the master node and the node 112 is the slave node, when the node 111 and the node 112 are in the Master-Slave Replication operation state, there can be multiple situations to switch to the master node independent operation state, for example:

[0039] In one case, if the node 112 cannot continue to provide gateway services due to failure or other reasons, the remaining node 111 will switch to the master node independent operation state to provide gateway services alone. Taking the write process as an example, in the master node independent operation state, the node 111 can obtain the write request from the user, store the specified data corresponding to the write request to the local cache of the node 111, and then store it to the storage system 12. Subsequently, after the node 112 recovers as a slave node of the gateway service, it can be switched back to the Master-Slave Replication operation state from the above master node independent operation state, at which time the node 111 needs to synchronize data to the node 112 to meet the requirement of data consistency.

[0040] In another case, if the node 111 cannot continue to provide the gateway service due to failure or other reasons, the remaining node 112 will be upgraded from a slave node to a master node to provide the gateway service independently in the master node independent running state. Taking the write process as an example, in the master node independent running state, the node 112 can obtain a write request from a user, store the specified data corresponding to the write request to the local cache of the node 112, and then store the specified data to the storage system 12. Subsequently, after the node 111 recovers, it will first serve as a slave node of the gateway service, so that the running state is switched from the master node independent running state to the master-slave replication running state, at which time the node 112 needs to synchronize data to the node 111 to meet the requirement of data consistency. After the data synchronization is completed, the node 111 can be upgraded to a master node, and the node 112 can be downgraded to a slave node.

[0041] In the related art, the understanding of data consistency is that the data copies held by different nodes can maintain the same or consistent state at a certain time point or after a data update operation. Based on this understanding, if the running state of the gateway service is switched from the master node independent running state to the master-slave replication running state, the master node needs to synchronize the full amount of data stored by the master node to the slave node to ensure that the data between the master node and the slave node can remain consistent, thereby meeting the requirement of data consistency. However, since the full amount of data needs to be synchronized, a large amount of bandwidth will be occupied, and the synchronization time is long.

[0042] To solve the above problems in the related art, the present disclosure provides an improved data synchronization scheme, which will be described in detail below in conjunction with the accompanying drawings.

[0043] FIG. 2 is a flowchart of a data synchronization method according to an example embodiment. As shown in FIG. 2, the method can include the following steps:

[0044] Step S202, determining the running state of the master-slave node providing the gateway service for the storage system.

[0045] The data synchronization method in this embodiment can run in the master node or in the slave node.

[0046] When the data synchronization method runs in the master node, the master node can determine the running state of the master-slave node by monitoring the status of the slave node on the basis of normal running of the master node to provide the gateway service, wherein: if the slave node is abnormal (such as normal communication but unable to work normally, or communication interruption, or other abnormal situations), the running state is determined as the master node independent running state; if the slave node runs normally to provide the gateway service, the running state is determined as the master-slave replication running state.

[0047] When the data synchronization method runs in the slave node, the slave node can determine that the running state of the master-slave node switches from the master node independent running state to the master-slave replication running state in the case that the slave node resumes to provide the gateway service after first providing the gateway service or failure. In combination with the foregoing description of the plurality of cases of the master node independent running state, it can be known that the identity of the slave node can be always unchanged, i.e., the slave node is originally a slave node, rather than a slave node formed by the degradation of the master node, or the slave node can also be a slave node formed by the degradation of the master node.

[0048] As an optional implementation, in the master-slave replication running state under the condition that the master node and the slave node are both normally running and are linked to each other, in the case that the master node acquires a write request, the master node writes the to-be-written data corresponding to the write request and to-be-written metadata into the master node cache storage module, wherein the to-be-written metadata includes storage state metadata information of the to-be-written data. Then, the master node synchronizes the to-be-written data and the to-be-written metadata to the slave node, so that the slave node writes the to-be-written data and the to-be-written metadata into the slave node cache storage module; and the master node synchronizes the to-be-written data and the to-be-written metadata to the storage system for persistence by the storage system. The write request is a request for requesting new writing of data in the storage system or modification of original data. When the master node receives the user write request, the master node writes the to-be-written data as cache data into the master node cache storage module according to the write request, and simultaneously, the master node also sends the to-be-written data to the slave node, waiting for the slave node to write the to-be-written data as cache data into the slave node cache storage module. Moreover, after acquiring the to-be-written data, the master node can generate to-be-written metadata corresponding to the to-be-written data, and also write the to-be-written metadata into the master node cache storage module. In addition, in the case that the user writes a plurality of to-be-written data in batches through the write request, the slave node can copy the to-be-written metadata corresponding to each to-be-written data from the master node in batches, and simultaneously, each to-be-written metadata can also be actively sent to the slave node by the master node. After acquiring the to-be-written metadata, the slave node can write the to-be-written metadata into the slave node cache storage module. The master node waits for the slave node to complete the copying of the to-be-written metadata and the to-be-written data, and determines that the synchronization process of the to-be-written metadata and the to-be-written data to the slave node is completed. After the synchronization between the master node and the slave node is completed, the file (i.e., the to-be-written data) updated by the write request can be uploaded and synchronized to the storage system asynchronously.

[0049] For example, as shown in FIG. 3, when the storage system is OSS and the gateway service is CSG (Cloud Storage Gateway), the above method can be implemented by the CSG file system process of the master node according to the following steps:

[0050] Step 31, receiving the user write request through the FUSE interface layer, wherein the FUSE interface layer can be an interface layer using a FUSE (Filesystem in Userspace) interface layer mechanism, and the FUSE interface layer can be combined with a tool (for example, nfs-ganesha (a user-mode NFS server implementation)) for implementing an NFS (Network File System) and SMB (Server Message Block) protocol process, so as to receive the request based on the NFS / SMB protocol.

[0051] Step 32, after the file system module in the master node forwards the user write request to the file system module in the master node through the FUSE interface layer, the file system module allocates or reserves the cache space according to its cache strategy, and sends a write cache data request (containing the to-be-written data to be cached and the reserved cache space) to the first data cache processing module according to the user write request.

[0052] Step 331, the first data cache replication module responds to the write cache data request to perform a cache data disk write operation on the to-be-written data, specifically: writing the to-be-written data as cache data into the reserved cache space in the master node cache storage module.

[0053] Step 332a, the first data cache processing module transmits the to-be-written data to the second data cache processing module. Optionally, the first data cache processing module can send a cache data replication request to the second data cache replication module of the slave node, so that the second data cache replication module can actively copy the to-be-written data from the first data cache replication module in response to the cache data replication request. In addition, the first data cache processing module can also actively send the to-be-written data to the second data cache replication module.

[0054] Step 332b, the second data cache processing module writes the obtained to-be-written data as cache data into the local slave node cache storage module by performing a cache data disk write operation.

[0055] Wherein, step 331 and 332a can be performed simultaneously, or step 331 can be performed before 332a, or step 332a can be performed before 331.

[0056] Step 34, the file system module of the master node also sends a write metadata request to the first metadata replication module of the master node, to instruct the first metadata replication module to cache the to-be-written metadata corresponding to the to-be-written data.

[0057] Step 351: After obtaining the metadata to be written corresponding to the data to be written, the first metadata replication module can write the metadata to be written to the master node cache storage module by performing a metadata write-to-disk operation.

[0058] In step 352a, the first metadata replication module transfers the metadata to be written to the second metadata replication module. Optionally, the first metadata replication module can send a metadata replication request to the second metadata replication module on the slave node, so that the second metadata replication module can respond to the metadata replication request and actively replicate the metadata to be written from the first metadata replication module. Alternatively, the first metadata replication module can actively send the metadata to be written to the second metadata replication module. Furthermore, when a user writes multiple pieces of data in batches through a write request, the second metadata replication module can retrieve the metadata corresponding to each piece of data from the first metadata replication module in batches.

[0059] Step 352b: After obtaining the metadata to be written, the second metadata replication module can write the metadata to be written to the slave node cache storage module by performing a metadata write-to-disk operation.

[0060] Steps 351 and 352a can be performed simultaneously, or step 351 can be executed before step 352a, or step 352a can be executed before step 351.

[0061] Step 36: After the first metadata replication module waits for the slave node to complete the replication of both the metadata to be written and the data to be written, it determines that the synchronization process of synchronizing the metadata to be written and the data to be written to the slave node is complete, generates corresponding synchronization process completion information, and sends the synchronization process completion information to the file system module so that the file system module can determine that after the synchronization between the master and slave nodes is completed, it can asynchronously upload and synchronize the file that the user wrote to request an update (i.e., the data to be written) to the storage system.

[0062] When a node fails or the connection between the master and slave nodes is broken, resulting in the master node running independently, the data writing process can be referred to the embodiment shown in Figure 3, which describes the implementation of writing the data corresponding to the user-written data and the storage status metadata information into the master node's local cache storage module. This will not be elaborated further here. However, due to a failure or a broken connection between the slave node and the master node, the master node cannot synchronize the data corresponding to the user-written data and the storage status metadata information to the slave node for caching. When the slave node recovers from the failure or the connection between the master and slave nodes is restored, the slave node can synchronize data from the master node, as described in subsequent steps S204-S206. This will not be elaborated further here.

[0063] If the primary node fails and the newly determined primary node operates independently after failover, the original primary node can no longer provide data writing services. Therefore, a failover will be performed to designate a secondary node as the new primary node. When a user writes data, the new primary node can refer to the description of the master-slave replication operation where both the primary and secondary nodes are running normally and interconnected, and write the data corresponding to the user's written data and the storage status metadata information into the primary node's local cache storage module. The detailed implementation method can be referred to the above content and will not be repeated here. However, since the secondary node converted from the primary node has failed, the new primary node cannot synchronize the data corresponding to the user's written data and the storage status metadata information to the new secondary node for caching. After the new secondary node recovers from the failure, the relevant content of subsequent steps S204-S206 can be referred to, and the new secondary node can synchronize data from the new primary node. This will not be repeated here.

[0064] Step S204: When the running state changes from the master node independent running state to the master-slave replication running state, the target data is filtered out from the local cache data of the master node. The target data has neither been successfully uploaded to the storage system nor successfully cached in the slave node.

[0065] In this embodiment, the target data refers to data that was only written to the cache storage module of the master node during its independent operation, but was not successfully uploaded to the storage system or cached on the slave nodes. The failure to upload to the storage system can be due to the following reasons: although stored in the storage system, the data in the storage system is abnormal (inconsistent with the data in the master node); there may be too much data to be synchronized to the storage system; or there may be a communication failure between the master node and the storage system, causing the data in the master node to not be uploaded to the storage system in a timely manner. Therefore, if the master node fails, the target data will inevitably become inaccessible. Furthermore, the form of the target data may vary depending on the storage system. For example, in an object storage system, data is stored as objects, each containing corresponding data and metadata. When the object storage system has a fixed-size or variable-size contiguous storage area for organizing and storing data, the target data can be stored as data blocks. In a file storage system, data is stored as files. Other storage methods are also possible, which will not be listed here. Reasons for unsuccessful caching to slave nodes can include, but are not limited to, the following categories: slave node failure, data errors in the master node's synchronization to slave nodes, and untimely data synchronization. Slave node failure can be caused by, but is not limited to: hardware or software malfunctions (e.g., CPU, memory, hard drive, network adapter, gateway service software, or operating system), leading to system crashes or failure to start, thus preventing the gateway service from running; slave node resource exhaustion, unable to handle more requests; and slave node overload, where the requests processed by the slave node exceed its processing capacity, resulting in slow system response or denial of service. Data errors in the master node's synchronization to slave nodes can include, but are not limited to: data loss or corruption during the synchronization process. Untimely data synchronization can include, but is not limited to: abnormal links between master and slave nodes (e.g., network latency, network terminal issues, or network congestion), preventing the master node from synchronizing data to the slave nodes; and excessive data requiring synchronization from the master node, causing delays in data synchronization from the master node to the slave nodes. As an optional implementation, step S204, which filters target data from the local cache data of the master node, includes the following steps: reading the metadata set maintained by the master node, which records the storage status metadata information corresponding to each piece of data, and the storage status metadata information is used to indicate whether the corresponding data has been successfully uploaded to the storage system and the node that has successfully cached the corresponding data; and determining the target data based on the storage status metadata information.

[0066] The master node stores at least one piece of data and storage status metadata information corresponding to each piece of data. The storage status metadata information can indicate the following situations: The data is stored on the master node, slave nodes, and the storage system. This situation can be the storage status metadata information updated when the master node successfully synchronizes the data to the slave nodes and the storage system. The data is stored on the master node and slave nodes, but not on the storage system. This situation can be the storage status metadata information updated when the master node successfully synchronizes the data to the slave nodes but fails to upload it to the storage system. The data is stored on the master node and the storage system, but not on the slave nodes. This situation can be the storage status metadata information updated when the master node successfully uploads the data to the storage system. The storage status metadata information is updated when the data is not successfully synchronized to the slave node. This can occur if the data is stored on the slave node and storage system, but not on the master node. This can happen if the master node (e.g., node 1) previously failed, and the slave node (e.g., node 2) was upgraded to master through failover and processed write requests, writing the corresponding data to the upgraded master node (i.e., node 2) and the storage system. If node 2 is downgraded from master to slave after node 1 recovers from the failure, the updated storage status metadata information is also updated if the data is stored on the master node but not on the slave node and storage system. This can happen if the master node failed to successfully upload the data to the storage system and synchronize it to the slave node. The metadata set maintained by the master node contains all storage status metadata information. Each piece of data can be associated with storage status metadata information through a unique identifier to ensure a one-to-one correspondence between storage status metadata information and data. For example, unique identifiers can include, but are not limited to, filenames, serial numbers, unique key values, etc. Simultaneously, this storage status metadata information can be used to characterize whether the corresponding data has been successfully uploaded to the storage system (i.e., whether the corresponding data has been successfully stored in the storage system). Furthermore, this storage status metadata information can also be used to characterize the nodes where the corresponding data has been successfully cached on the master and slave nodes. When the storage status metadata information can characterize the aforementioned storage status, it is possible to determine, through the storage status metadata information corresponding to each local cached data, target data that has neither been successfully uploaded to the storage system nor successfully cached on slave nodes. Through the method of this embodiment, by using storage status metadata information to indicate whether the corresponding data has been successfully uploaded to the storage system and successfully cached on nodes, the storage status metadata information can accurately characterize the storage status of the corresponding data, thereby effectively ensuring that the determined target data meets the requirement of neither being successfully uploaded to the storage system nor successfully cached on slave nodes.

[0067] Optionally, any storage state metadata information in the storage state metadata information set may include status variables indicating whether the corresponding data has been successfully uploaded to the storage system and the nodes that have successfully cached it. The values ​​of these status variables can then characterize whether the data corresponding to any storage state metadata information has been successfully uploaded to the storage system and the nodes that have successfully cached the corresponding data. Specifically, the storage state metadata information corresponding to any data may include a first status variable and a second status variable. For the first status variable, a first value indicates that any data has been successfully uploaded to the storage system, and a second value indicates that any data has not been successfully uploaded to the storage system. The second status variable includes values ​​indicating the nodes that have successfully cached any data.

[0068] For example, take the following storage status metadata information corresponding to Chunk0 with a sequence number of 0 as an example: C0{IsClean: True, DateNodeIds: (1, 2)}, where IsClean can be the aforementioned first status variable. When the first status variable is the first value, i.e., True, it indicates that the data corresponding to the storage status metadata information has been uploaded to the storage system. When the first status variable is the second value, such as False, it indicates that the data corresponding to the storage status metadata information has not been uploaded to the storage system. Furthermore, each value in the second state variable can be used to indicate the node that has successfully cached any data corresponding to each value. Taking C0 as an example, DateNodeIds can be the second state variable. In C0, the value of DateNodeIds is (1, 2). If 1 is used to indicate the value of the master node and 2 is used to indicate the value of the slave node, then since C0 includes the values ​​1 and 2, the second state variable can be used to indicate that the corresponding data Chunk0 has been successfully cached to both the master node and the slave node. If the value of DateNodeIds in another storage state metadata information CX{IsClean:False,DateNodeIds:(1)} is (1), then it indicates that the data ChunkX corresponding to CX has only been successfully cached to the master node and has not been successfully cached to the slave node. Through the method of this embodiment, by setting the first state variable and the second state variable to respectively represent whether the corresponding data has been successfully uploaded to the storage system and the node that has been successfully cached, the storage state metadata information can be modified in real time according to the storage state so that it can be used to represent the accurate storage state of the corresponding data.

[0069] Figure 4 illustrates three different file states:

[0070] 1. C1 indicates that Chunk1 in file 1 is clean data (i.e., Chunk1 is stored on OSS), and Chunk1 is consistent on both the master and slave nodes. C2 indicates that Chunk2 in file 1 has been modified, and Chunk2 has been modified, and Chunk2 is consistent on both the master and slave nodes.

[0071] 2. C3-C6 indicate that file 2 is a clean file (i.e., file 2 is stored on OSS), and has been partially read on both the master and slave nodes. This means that a portion of the data of this file exists in the cache on both the master and slave nodes. Specifically, C3 indicates that data Chunk3 has been read on the slave node, C4 indicates that data Chunk4 has been read on the master node, C5 indicates that data Chunk5 is clean and that data Chunk5 is consistent on both the master and slave nodes, and C6 indicates that Chunk6 has been read on the master node.

[0072] 3. File 3 is a dirty file (i.e., the latest file 3 is not stored on OSS), where:

[0073] a. C8 and C10 are used to indicate data Chunk 8 and data Chunk 10, respectively, which are the data written to the master node for single-node service.

[0074] b.C7 is used to indicate that the data Chunk 7 is dirty and exists on both the master and slave nodes.

[0075] c.C9 is used to indicate that data Chunk 9 is Clean and that data Chunk 9 is consistent across the master node and OSS.

[0076] Therefore, the data in Chunk 8 and Chunk 10, since it is only stored on the master node, cannot be retrieved from other devices if the master node fails. Thus, Chunk 8 and Chunk 10 need to be resynchronized and replicated to the slave nodes. While Chunks 4, 6, and 9 in the diagram only have data on the master node, they are clean (meaning the data Chunks 4, 6, and 9 are stored on OSS). Therefore, they do not need to be replicated to the slave nodes. If the master node fails and a failover occurs, this data can be downloaded again from OSS.

[0077] As mentioned earlier, the storage status metadata information corresponding to any data may include a first status variable and a second status variable, indicating whether the data has been successfully uploaded to the storage system and which node has completed caching, respectively. Accordingly, when the master node caches any data locally, it can add a value corresponding to the master node to the second status variable; when any data is successfully uploaded to the storage system by the master node, the master node can replace the first value of the first status variable with the second value; when any data is successfully cached on a slave node, the master node can add a value corresponding to the slave node to the second status variable. In other words, for any data, when any data is uploaded to the storage system by the master node, the first state variable is replaced with the second value. That is, when any data has been successfully uploaded to the storage system, it means that any data is no longer dirty data. Therefore, the value of the corresponding first state variable also needs to be able to represent that any data has been stored in the storage system. Thus, the first state variable needs to be replaced with the second value. For example, taking the aforementioned CX as an example, when the data ChunkX corresponding to CX is successfully uploaded to the storage system, the value of its first state variable, IsClean, can be replaced with the second value (i.e., False) instead of the first value (i.e., True). For any data, when any data is synchronized from the master node to the slave node, the value corresponding to the slave node is added to the value of the second state variable. That is, when any data has been successfully synchronized to the slave node, it means that the data is currently stored in both the master node and the slave node. Therefore, the value of the corresponding second state variable also needs to be able to represent that the data has been stored in the slave node. Thus, the value corresponding to the slave node needs to be added to the second state variable. For example, taking the aforementioned CX as an example, when the data ChunkX corresponding to CX is successfully synchronized to the slave node, the value of its second state variable, DateNodeIds, can be increased by the value corresponding to the slave node (for example, the value corresponding to the slave node is 2) on the basis of the original value (i.e., (1)) to obtain the new value of the second state variable (i.e., (1, 2)). In this embodiment, by replacing the value of the first state variable when any data is uploaded to the storage system from the master node, the value of the first state variable can be used to intuitively represent whether any data has been uploaded to the storage system. By increasing the value of the second state variable when any data is successfully cached from the master node to the slave node, the value of the second state variable can be used to intuitively represent whether any data has been uploaded to the slave node. The method of this embodiment can directly represent the storage status of any data from the storage status metadata information, which can facilitate the quick selection of target data during subsequent data synchronization.

[0078] As an optional implementation, when the target data is selected by the master node, the master node directly reads the locally maintained metadata set. That is, in this embodiment, when steps S202 to S206 are implemented by the master node, the target data is also selected by the master node. In this case, the master node can directly read the metadata from its local cache disk to obtain the metadata set. Furthermore, to ensure metadata consistency between the master and slave nodes, the master node can synchronize its own metadata to the slave node. The master node can determine whether it needs to fully synchronize its own metadata to the slave node by comparing the metadata in the master and slave nodes. When the master node determines that any metadata from the master node is missing from the slave node, it determines that a full synchronization of metadata to the slave node is necessary. A full metadata synchronization task can be executed, and the full metadata set in the master node can be obtained by copying all SST (Sorted String Table) files from the cache disk in the master node through a checkpoint. This full metadata set is then synchronized to the slave node. When the master node determines that it does not need to fully synchronize metadata to the slave node, it can determine the target master node metadata that needs to be synchronized to the slave node based on the difference between the sequence number of the master node metadata (i.e., the storage state metadata information in the master node) and the sequence number of the slave node metadata (i.e., the storage state metadata information in the slave node), and synchronize the target master node metadata to itself. The sequence number of the target master node metadata does not exist in the metadata of any of the slave nodes.

[0079] When the target data is selected by the slave node, the slave node synchronizes its own metadata set based on the metadata set maintained by the master node and reads the synchronized metadata set. In other words, in this embodiment, when steps S202 to S206 are implemented by the slave node, the target data is selected by the slave node. Optionally, the slave node can determine whether a full synchronization of the master node's metadata is needed by comparing the metadata in the master and slave nodes. If the slave node determines that it does not contain any metadata from the master node, it determines that a full synchronization of metadata from the master node is needed. It can then execute a full metadata synchronization task and obtain the metadata set from the master node by copying all SST files from the master node through checkpointing. When the slave node determines that a full synchronization of metadata from the master node is not needed, it can determine the target master node metadata to be synchronized to the slave node based on the sequence number of the master node's metadata and the sequence number of the slave node's metadata, and synchronize the target master node metadata to itself.

[0080] As an optional implementation, step S204, which filters target data from the data locally cached on the master node, can also be achieved through the following steps: obtaining a first metadata set maintained by the master node, a second metadata set maintained by the slave node, and a third metadata set maintained by the storage system, respectively. The first metadata set records the storage status metadata information corresponding to each piece of data cached on the master node; the second metadata set records the storage status metadata information corresponding to each piece of data cached on the slave node; and the third metadata set records the storage status metadata information corresponding to each piece of data stored in the storage system. By comparing the first, second, and third metadata sets, the storage status of each piece of data on the master node, slave node, and storage system is determined, and the target data is filtered out from all the data based on the storage status. This step can also be implemented by either the master node or the slave node: the first metadata set can be obtained by the first data cache replication module in the master node from the master node's cache storage module; the second metadata set can be obtained by the second data cache replication module in the slave node from the slave node's cache storage module; and the third metadata set can be obtained by either the master node or the slave node from the storage system. In this embodiment, when the method is implemented by the master node, the master node can read the third metadata set from the storage system. When the method is implemented by the slave node, the slave node can read the third metadata set from the storage system.

[0081] After obtaining the first, second, and third metadata sets, the storage status of the data already stored on the master node, slave nodes, and storage system can be determined through the storage status metadata information in each metadata set and the one-to-one correspondence between metadata and data. Based on this storage status, the target data that is only cached on the master node and has not been successfully uploaded to the storage system or cached on the slave nodes can be identified. For example, when the first metadata set contains storage status metadata information corresponding to the following sequence numbers: C101, C102, C103, C104, C106; the second metadata set contains all storage status metadata information corresponding to the following sequence numbers: C101, C102, C103; and the third metadata set contains all storage status metadata information corresponding to the following sequence numbers: C101, C103, C105, C106, it can be determined that the target data cached only on the master node is the data indicated by storage status metadata information C104.

[0082] By using the method of this embodiment, it is only necessary to obtain the metadata set of the storage status of the master node, slave node and storage system to determine the storage status of each data in the master node, slave node and storage system. Based on the storage status, the quantity of each data stored in the master node, slave node and storage system can be determined, and then the target data can be determined from all the data contained in the master node, slave node and storage system based on the storage status.

[0083] Step S206: Start the data cache synchronization task to synchronize the target data from the master node to the slave node.

[0084] After the target data is determined, the target data can be synchronized from the master node to the slave node through a data caching synchronization task between the master node and the slave node.

[0085] Optionally, the master node can initiate the data cache synchronization task. The specific implementation method is as follows: After the master node determines the target data to be transmitted to the slave node, it can obtain the target data from the local master node cache storage module through the first data cache replication module on the master node, and then actively transmit the target data to the second data cache replication module on the slave node. The second data cache replication module can write the target data into the slave node cache storage module on the slave node.

[0086] Alternatively, the data cache synchronization task can be initiated by the slave node. The specific implementation method is as follows: After the slave node determines the target data to be obtained, it can actively send a data acquisition request to obtain the target data from the master node through the second data cache replication module on the slave node. After the second data cache replication module obtains the data acquisition request, it can read the target data from the cache storage module of the master node. Then, the first data cache replication module transmits the acquired target data to the second data cache replication module.

[0087] Furthermore, in any of the above embodiments, the first data cache copying module can actively send the target data to the second data cache copying module, or the second data cache copying module can copy it from the first data cache copying module.

[0088] Furthermore, for any corresponding data and storage status metadata information, if either the data or the latest storage status metadata information (the second variable includes the value of the target storage area) fails to be successfully synchronized to the target storage area (i.e., the storage system or slave node), both the data and the latest storage status metadata information will fail to synchronize. The latest storage status metadata information will be rolled back to the storage status metadata information before synchronization, meaning the second variable will delete the value of the target storage area. In summary, any data and storage status metadata information will only be successfully synchronized if both are successfully synchronized. This means that there is atomicity between the data synchronization process and the metadata synchronization process. If this setting is not followed, and if data synchronization is successful but storage status metadata information is not successfully synchronized, the target area will not have storage status metadata information corresponding to the data, making it impossible to retrieve the corresponding data in the target area later. If storage status metadata information is successfully synchronized but data synchronization is unsuccessful, the storage status metadata information already includes the target storage area value, meaning it indicates that the data is stored in the target storage area. Consequently, the storage status metadata information cannot accurately represent the storage status of the data.

[0089] The method in this embodiment ensures high availability and data consistency for a given data by storing it in at least two of the master node, slave nodes, and storage system (i.e., at least two of the master node, slave nodes, and storage system). Specifically, when a data is stored in the master node and storage system but not in slave nodes, during data retrieval, if the master node is functioning correctly, it can be read directly from the master node. Furthermore, regardless of whether the storage system or a slave node fails, the master node can still respond normally and retrieve the data. Additionally, even if the master node fails, a slave node can retrieve the data from the storage system when it is promoted to master and responds to a read request. Similarly, when a data is stored in both the master node and slave nodes but not in the storage system, during data retrieval, if the master node is functioning correctly, it can be read directly from the master node. Furthermore, regardless of whether the storage system or a slave node fails, the master node can still respond normally and retrieve the data. Additionally, even if the master node fails, a slave node can retrieve the data directly from the new master node when it is promoted to master and responds to a read request. Therefore, when the operating state between the master and slave nodes changes from the master node independent operating state to the master-slave replication operating state, it is only necessary to identify the target data that has not been successfully uploaded to the storage system or successfully cached on the slave node from the master node, and cache the target data on the slave node to meet the data consistency requirements of this disclosure. For other data, such as data that is not on the slave node but is already stored on the master node and in the storage system, there is no need to synchronize it to the slave node. Thus, the amount of data synchronized from the master node to the slave node can be effectively reduced. In this way, while ensuring the high availability of storage services and preventing the loss of target data, the bandwidth consumption of data synchronization can be reduced and the data synchronization time can be reduced.

[0090] As an optional implementation, the method further includes: when the master node receives a read request from the reader, the master node responds to the read request and determines whether the data to be read meets the target data requirements based on the storage status metadata information of the data to be read. The read request requests the data to be read, and the target data requirements indicate that the corresponding data exists in the master node's cache storage module and that the data is reliable in the master node's cache storage module. If the data to be read does not meet the target data requirements, the master node retrieves the data to be read from the storage system, returns the cached data to be read from the object storage module to the reader, and stores the data to be read only on the master node among the master and slave nodes. When the master node receives a read request from the reader to read local cached data, the master node first determines the storage status metadata information corresponding to the data to be read based on the read request, then determines whether the data to be read exists in the master node's local cache module based on the storage status metadata information. If the data to be read exists in the master node's local cache module, the master node directly reads the data from the master node's local cache module, retrieves the data to be read, and returns it to the reader. If the master node determines that the storage status metadata information of the data to be read is invalid (for example, the master node does not store the data to be read, but the storage status metadata information indicates that the data to be read is Clean (i.e., stored in the storage system)), specifically, the master node can obtain the storage status metadata information of the data to be read locally. If the storage status metadata information indicates that the master node has cached the data to be read locally, but the master node cannot find the data to be read in the master node's cache storage module, then the master node can determine that the storage status metadata information of the data to be read is invalid, and can determine that the data to be read is unreliable in the master node's local cache module. Therefore, it is necessary to obtain the data to be read from the storage system. Since the data to be read already exists in the storage system, it is only necessary to store it locally on the master node, without storing it on the slave node. Therefore, it can effectively reduce the network bandwidth usage and reduce the time required for master-slave synchronization. Furthermore, even if a failover occurs and the newly determined master node does not store the data to be read, the correct data can still be obtained from the storage system using the method of this embodiment.

[0091] As shown in Figure 5, this paper presents an implementation method for users to read data from the master node when the storage system is OSS, the gateway service is CSG, and the data read is in data blocks.

[0092] Step 51: The master node obtains the user read request from the user terminal through the FUSE interface layer.

[0093] Step 52: The file system module generates a Chunk status information read request based on the user read request sent by the FUSE interface layer, and sends the Chunk status information read request to the metadata module.

[0094] Step 53: In response to the request to read Chunk status information, the metadata module reads the Chunk status information (i.e., storage status metadata information) corresponding to the data block to be read from the master node cache storage module by performing a disk read operation.

[0095] Step 54: The metadata module determines whether the data block to be read exists in the master node cache storage module (i.e., the master node's local cache disk) based on the Chunk status information. If the data block exists in the master node cache storage module, it can be directly read from the master node's local cache disk through the data cache module to obtain the data block, and then returned to the user by executing step 59. If the metadata module determines that the Chunk is invalid based on the Chunk status information (e.g., there is no data in the Chunk, but the Chunk status information indicates that the data block to be read is Clean (i.e., stored in OSS)), it can be determined that the data block to be read is unreliable in the master node cache storage module. Therefore, it is necessary to obtain the data block to be read from the OSS storage bucket bound to the master node by executing step 56, and the Chunk status information needs to be updated synchronously so that the updated Chunk status information can be used to indicate that the data block to be read is stored in both the master node and OSS.

[0096] Step 55: The metadata module writes the updated Chunk status information to the master node cache storage module by performing a disk write operation.

[0097] Step 56: The file system module downloads the data blocks to be read from OSS.

[0098] Step 57: The file system module sends a populate local cache instruction to the data cache module, instructing the data cache module to populate the local cache with the data blocks to be read downloaded from OSS.

[0099] In step 58, in response to the received local cache fill instruction, the data caching module caches the data blocks downloaded from OSS to be read into the master node cache storage module by performing a disk write operation. Steps 55 and 56 can be performed simultaneously, or step 55 can be executed before step 56, or step 56 can be executed before step 55. Furthermore, steps 55 and steps 56-58 are atomic, meaning they can either succeed simultaneously or fail simultaneously.

[0100] Step 59: The file system module sends the data block to be read to the user end through the FUSE interface layer.

[0101] Since the data block to be read already exists in the OSS storage bucket, and after executing steps 51 to 58, the data block to be read is also cached in the master node cache storage module, the master and slave nodes and most of the OSS have stored the data block to be read. This reduces the bandwidth usage of data synchronization and the time consumption of data synchronization while ensuring the high availability of storage services and preventing the loss of target data.

[0102] Figure 6 is a flowchart of a data synchronization method for a cloud storage gateway according to an exemplary embodiment. The cloud storage gateway includes a master node and a slave node for providing cloud storage gateway services to an object storage system. As shown in Figure 6, the method may include the following steps:

[0103] Step S602: Determine the running status of the master and slave nodes that provide cloud storage gateway services for the object storage system;

[0104] Step S604: When the running status changes from the master node independent running status to the master-slave replication running status, the target data is filtered out from the data cached locally on the master node. The master-slave replication running status indicates that the master node and the slave nodes connected to the master node are running normally, and the target data has neither been successfully uploaded to the storage system nor successfully cached on the slave nodes.

[0105] Step S606: Start the data cache synchronization task to synchronize the target data from the master node to the slave node.

[0106] The object storage system in this embodiment is an optional storage system type in the technical scenario of cloud storage gateway in the aforementioned embodiments. Therefore, the relevant implementation method for determining the faulty storage node in this embodiment can refer to the relevant implementation methods of steps S202 to S206 in the aforementioned embodiments, and will not be repeated here.

[0107] As an optional implementation, the target data is filtered from the local cache data of the master node, including:

[0108] Read the metadata set maintained by the master node. The metadata set records the storage status metadata information corresponding to each piece of data. The storage status metadata information is used to indicate whether the corresponding data has been successfully uploaded to the object storage system and the node that has successfully cached the corresponding data.

[0109] The target data is determined based on the stored state metadata information.

[0110] The implementation method in this embodiment can refer to the implementation method of filtering target data from the local cache data of the master node in the previous embodiment, and will not be repeated here.

[0111] As an optional implementation, the storage state metadata information corresponding to any data includes:

[0112] The first state variable, when it is in the first value, indicates that any data has been successfully uploaded to the object storage system; when it is in the second value, it indicates that any data has not been successfully uploaded to the storage system.

[0113] The second state variable has a value that indicates whether any data has been successfully cached.

[0114] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0115] As an optional implementation, the method further includes:

[0116] If any data is successfully uploaded to the object storage system from the master node, the first state variable will be replaced with the second value.

[0117] If any data is successfully cached to the slave node, the value corresponding to the slave node is added to the value of the second state variable.

[0118] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0119] As an optional implementation, the metadata set maintained by the master node is read, including:

[0120] When the target data is selected by the master node, the master node reads the metadata set maintained locally;

[0121] When the target data is filtered out by the slave node, the slave node synchronizes its own metadata set with the metadata set maintained by the master node and reads the synchronized metadata set.

[0122] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0123] As an optional implementation, the target data is filtered from the data cached locally on the master node, including:

[0124] The first metadata set maintained by the master node, the second metadata set maintained by the slave node, and the third metadata set maintained by the object storage system are obtained respectively. The first metadata set records the storage status metadata information corresponding to each data in all data cached on the master node, the second metadata set records the storage status metadata information corresponding to each data cached on the slave node, and the third metadata set records the storage status metadata information corresponding to each data stored on the storage system.

[0125] By comparing the first metadata set, the second metadata set, and the third metadata set, the storage status of each data in the master node, slave node, and object storage system is determined, and the target data is selected from all data based on the storage status.

[0126] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0127] As an optional implementation, the method further includes:

[0128] When the master node receives a read request from the reading end, the master node responds to the read request and determines whether the data to be read meets the target data requirements based on the storage status metadata information of the data to be read. The read request is used to request the data to be read, and the target data requirements are used to indicate that the corresponding data exists in the master node cache storage module and that the data is reliable in the master node cache storage module.

[0129] If the data to be read does not meet the target data requirements, the master node retrieves the data to be read from the object storage system, returns the cached data to be read from the object storage system to the reading end, and stores the data to be read only on the master node among the master and slave nodes.

[0130] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0131] As an optional implementation, the method further includes:

[0132] When the master node receives a write request from the writing end, the master node writes the data to be written and the corresponding metadata to be written to the master node cache storage module. The metadata to be written includes the storage status metadata information of the data to be written.

[0133] The master node synchronizes the data to be written and the metadata to be written to the slave node, so that the slave node writes the data to be written and the metadata to be written to the slave node cache storage module; and the master node synchronizes the data to be written and the metadata to be written to the storage system for persistence by the storage system.

[0134] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0135] As shown in Figure 7, a method for implementing master-slave synchronization after data inconsistency occurs between the master and slave nodes is provided, applying any of the aforementioned embodiments:

[0136] When a slave node disconnects or the master node fails and is transferred, the new master node switches to a single-node service mode. At this time, the master and slave nodes operate independently. User-written data is only successfully written to the master node's cache disk (an optional master node cache storage module) and cannot be replicated to the slave nodes. Therefore, the data on the master and slave nodes is inconsistent. When the slave node comes back online, i.e., the master and slave nodes return to a coordinated operating state, the data on the master and slave nodes needs to be synchronized to restore a synchronized master-slave state. After starting, the slave node continuously pulls and replicates metadata update log data from the master node. Simultaneously, the master node begins executing background synchronization tasks, the specific process of which includes:

[0137] Step 71: The master node determines whether it needs to fully synchronize metadata (i.e., storage status metadata information) to the slave node. If yes, proceed to step 72; otherwise, proceed to step 73.

[0138] Step 72: The master node performs a full metadata synchronization task. It can obtain the full metadata (i.e., the metadata set) from the master node's cache disk by copying all SST files through checkpointing, and then synchronizes the full metadata to the slave nodes. It also notifies the slave nodes to perform metadata replication.

[0139] Step 73: After the master node performs a full metadata synchronization task, some metadata may fail to be successfully synchronized to the slave node or newly generated metadata may not be synchronized to the slave node. In this case, in order to ensure that the metadata of the master and slave nodes are completely consistent, the master node can determine the target master node metadata that failed to be synchronized to the slave node or was newly generated in the full synchronization task based on the difference in sequence numbers between the master node metadata (i.e., the metadata in the master node) and the slave node metadata (i.e., the metadata in the slave node). The master node can also determine whether the metadata has been synchronized in real time based on the difference in sequence numbers. When it is determined that the metadata has been synchronized, step 74 can be executed. When it is determined that the metadata has not been synchronized, step 73 is executed again until the metadata is synchronized. In addition, when the master node determines that it does not need to fully synchronize metadata to the slave node, in order to ensure that the metadata of the master and slave nodes are completely consistent, the master node can also determine the target master node metadata (i.e., metadata that does not exist in the slave node) that needs to be synchronized to the slave node based on the difference in the sequence number of the master node metadata and the slave node metadata. The master node can also determine whether the metadata synchronization is complete in real time based on the difference in the sequence number. When it is determined that the metadata synchronization is complete, step 74 can be executed. When it is determined that the metadata synchronization is not complete, step 73 is executed again until the metadata synchronization is complete.

[0140] Step 74: The tag metadata synchronization is complete, and the system enters synchronous write mode.

[0141] Step 75: Start the data cache synchronization task, which is used to synchronize cached data to slave nodes.

[0142] Step 76: Traverse the metadata of the cached data blocks to obtain the metadata of each data block, and then determine the status information of each data block. Based on the status information of the data blocks, the data blocks can be copied to the slave nodes as needed. That is, among all data blocks, the target data blocks that have not been successfully uploaded to the storage system or successfully cached to the slave nodes are identified, and the target data blocks are copied to the slave nodes.

[0143] Step 77: After copying is complete, determine whether any exceptions occurred during the copying process. If no exceptions occurred, the synchronization is considered complete; otherwise, the synchronization is considered to have failed and a retry is initiated.

[0144] It also synchronizes the target master node's metadata to itself, where the sequence number of the target master node's metadata does not exist in the metadata of any of the slave nodes.

[0145] Figure 8 is a schematic structural diagram of a device provided in an exemplary embodiment. Referring to Figure 8, at the hardware level, the device includes a processor 802, an internal bus 804, a network interface 806, a memory 808, and a non-volatile memory 810, and may also include other hardware required for its functions. One or more embodiments of this disclosure can be implemented in software, for example, the processor 802 reads the corresponding computer program from the non-volatile memory 810 into the memory 808 and then runs it. Of course, in addition to software implementation, one or more embodiments of this disclosure do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0146] Please refer to Figure 9. The data synchronization device can be applied to the device shown in Figure 8 to implement the technical solution of this disclosure. The data synchronization device may include:

[0147] The running status determination module 91 is used to determine the running status of the master and slave nodes that provide gateway services to the storage system;

[0148] The filtering module 92 is used to filter target data from the local cache data of the master node when the running status is switched from the master node independent running status to the master-slave replication running status. The target data is neither successfully uploaded to the storage system nor successfully cached in the slave node.

[0149] Synchronization module 93 is used to start a data cache synchronization task to synchronize the target data from the master node to the slave node.

[0150] As an optional implementation, the screening module 92 includes:

[0151] The read unit is used to read the metadata set maintained by the master node. The metadata set records the storage status metadata information corresponding to each piece of data. The storage status metadata information is used to indicate whether the corresponding data has been successfully uploaded to the storage system and the node that has successfully cached the corresponding data.

[0152] The determination unit is used to determine the target data based on the stored state metadata information.

[0153] As an optional implementation, the storage state metadata information corresponding to any data includes:

[0154] The first state variable indicates that any data has been successfully uploaded to the storage system when it is in the first value, and that any data has not been successfully uploaded to the storage system when it is in the second value.

[0155] The second state variable has a value that indicates whether any data has been successfully cached.

[0156] As an optional implementation, an editing module is also included, the editing module being configured to:

[0157] If any data is successfully uploaded to the storage system by the master node, the first state variable will be replaced with the second value.

[0158] If any data is successfully cached to the slave node, the value corresponding to the slave node is added to the value of the second state variable.

[0159] As an optional implementation, the reading unit is configured as follows:

[0160] When the target data is selected by the master node, the master node reads the metadata set maintained locally;

[0161] When the target data is filtered out by the slave node, the slave node synchronizes its own metadata set with the metadata set maintained by the master node and reads the synchronized metadata set.

[0162] As an optional implementation, the filtering module 92 is configured as follows:

[0163] The first metadata set maintained by the master node, the second metadata set maintained by the slave node, and the third metadata set maintained by the storage system are obtained respectively. The first metadata set records the storage status metadata information corresponding to each data cached on the master node, the second metadata set records the storage status metadata information corresponding to each data cached on the slave node, and the third metadata set records the storage status metadata information corresponding to each data stored on the storage system.

[0164] By comparing the first metadata set, the second metadata set, and the third metadata set, the storage status of each data in the master node, slave node, and storage system is determined, and the target data is selected from the local cache data based on the storage status.

[0165] As an optional implementation, a reading module is also included, configured to:

[0166] When the master node receives a read request from the reading end, the master node responds to the read request and determines whether the data to be read meets the target data requirements based on the storage status metadata information of the data to be read. The read request is used to request the data to be read, and the target data requirements are used to indicate that the corresponding data exists in the master node cache storage module and is trusted in the master node cache storage module.

[0167] If the data to be read does not meet the requirements of the target data, the master node retrieves the data to be read from the storage system, returns the data to be read from the storage system to the reading end, and stores the data to be read only on the master node in the master-slave node hierarchy.

[0168] As an optional implementation, a writing module is also included, configured to:

[0169] When the master node receives a write request from the writing end, the master node writes the data to be written and the corresponding metadata to be written to the master node cache storage module. The metadata to be written includes the storage status metadata information of the data to be written.

[0170] The master node synchronizes the data to be written and the metadata to be written to the slave node, so that the slave node writes the data to be written and the metadata to be written to the slave node cache storage module; and the master node synchronizes the data to be written and the metadata to be written to the storage system for persistence by the storage system.

[0171] Please refer to Figure 10. The data synchronization device of the cloud storage gateway can be applied to the device shown in Figure 7 to realize the technical solution of this disclosure. The cloud storage gateway includes a master node and slave nodes for providing cloud storage gateway services to the object storage system; the data synchronization device of the cloud storage gateway may include:

[0172] The cloud storage gateway status determination module 101 is used to determine the operating status of the master and slave nodes that provide cloud storage gateway services to the object storage system.

[0173] The filtering module 102 is used to filter target data from the local cache data of the master node when the running state is switched from the master node independent running state to the master-slave replication running state. The target data is neither successfully uploaded to the storage system nor successfully cached in the slave node.

[0174] Synchronization module 103 is used to start a data cache synchronization task to synchronize the target data from the master node to the slave node.

[0175] As an optional implementation, the screening module 102 includes:

[0176] The read unit is used to read the metadata set maintained by the master node. The metadata set records the storage status metadata information corresponding to each piece of data. The storage status metadata information is used to indicate whether the corresponding data has been successfully uploaded to the object storage system and the node that has successfully cached the corresponding data.

[0177] The determination unit is used to determine the target data based on the stored state metadata information.

[0178] The implementation method in this embodiment can refer to the implementation method of filtering target data from the local cache data of the master node in the previous embodiment, and will not be repeated here.

[0179] As an optional implementation, the storage state metadata information corresponding to any data includes:

[0180] The first state variable, when it is in the first value, indicates that any data has been successfully uploaded to the object storage system; when it is in the second value, it indicates that any data has not been successfully uploaded to the storage system.

[0181] The second state variable has a value that indicates whether any data has been successfully cached.

[0182] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0183] As an optional implementation, an editing module is also included, the editing module being configured to:

[0184] If any data is successfully uploaded to the object storage system from the master node, the first state variable will be replaced with the second value.

[0185] If any data is successfully cached to the slave node, the value corresponding to the slave node is added to the value of the second state variable.

[0186] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0187] As an optional implementation, the reading unit is configured as follows:

[0188] When the target data is selected by the master node, the master node reads the metadata set maintained locally;

[0189] When the target data is filtered out by the slave node, the slave node synchronizes its own metadata set with the metadata set maintained by the master node and reads the synchronized metadata set.

[0190] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0191] As an optional implementation, the filtering module 102 is configured as follows:

[0192] The first metadata set maintained by the master node, the second metadata set maintained by the slave node, and the third metadata set maintained by the object storage system are obtained respectively. The first metadata set records the storage status metadata information corresponding to each piece of data cached on the master node, the second metadata set records the storage status metadata information corresponding to each piece of data cached on the slave node, and the third metadata set records the storage status metadata information corresponding to each piece of data stored on the storage system.

[0193] By comparing the first metadata set, the second metadata set, and the third metadata set, the storage status of each data in the master node, slave node, and object storage system is determined, and the target data is selected from the local cache data of the master node based on the storage status.

[0194] The implementation methods in this embodiment can refer to the implementation methods in the foregoing embodiments, and will not be repeated here.

[0195] As an optional implementation, a reading module is also included, configured to:

[0196] When the master node receives a read request from the reading end, the master node responds to the read request and determines whether the data to be read meets the target data requirements based on the storage status metadata information of the data to be read. The read request is used to request the data to be read, and the target data requirements are used to indicate that the corresponding data exists in the master node cache storage module and is trusted in the master node cache storage module.

[0197] If the data to be read does not meet the requirements of the target data, the master node retrieves the data to be read from the object storage system, returns the data to be read from the object storage system to the reading end, and stores the data to be read only on the master node in the master-slave node hierarchy.

[0198] As an optional implementation, a writing module is also included, configured to:

[0199] When the master node receives a write request from the writing end, the master node writes the data to be written and the corresponding metadata to be written to the master node cache storage module. The metadata to be written includes the storage status metadata information of the data to be written.

[0200] The master node synchronizes the data to be written and the metadata to be written to the slave node, so that the slave node writes the data to be written and the metadata to be written to the slave node cache storage module; and the master node synchronizes the data to be written and the metadata to be written to the storage system for persistence by the storage system.

[0201] Based on the same concept as the methods described above, this disclosure also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor executes the executable instructions to implement the steps of the method as described in any of the above embodiments.

[0202] Based on the same concept as the methods described above, this disclosure also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0203] Based on the same concept as the methods described above, this disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

Claims

1. A data synchronization method, wherein, The method comprises: determining the running state of a master-slave node providing gateway service for a storage system; in the case where the running state is switched from a master node independent running state to a master-slave replication running state, screening target data from the local cache data of the master node, wherein the target data has neither been successfully uploaded to the storage system nor successfully cached to a slave node; starting a data cache synchronization task to synchronize the target data from the master node to the slave node.

2. The method of claim 1, wherein, The screening of the target data from the local cache data of the master node comprises: reading a metadata set maintained by the master node, wherein the metadata set records the storage state metadata information corresponding to each data, and the storage state metadata information is used to indicate whether the corresponding data has been successfully uploaded to the storage system and the node that has successfully cached the corresponding data; determining the target data according to the storage state metadata information.

3. The method of claim 2, wherein, The storage state metadata information corresponding to any data comprises: a first state variable, wherein when the first state variable is in a first value, it indicates that the any data has been successfully uploaded to the storage system, and when the first state variable is in a second value, it indicates that the any data has not been successfully uploaded to the storage system; a second state variable, wherein the value of the second state variable comprises a value indicating the node that has successfully cached the any data.

4. The method of claim 3, wherein, The method further comprises: in the case where the any data has been successfully uploaded to the storage system by the master node, replacing the first state variable from the first value to the second value; in the case where the any data has been successfully cached to the slave node, adding a value corresponding to the slave node to the value of the second state variable.

5. The method according to any one of claims 2 to 4, wherein, The reading of the metadata set maintained by the master node comprises: in the case where the target data is screened by the master node, the master node reads the metadata set maintained locally; in the case where the target data is screened by the slave node, the slave node synchronizes the metadata set maintained by itself according to the metadata set maintained by the master node, and reads the synchronized metadata set.

6. The method according to any one of claims 1 to 5, wherein, The screening of the target data from the local cache data of the master node comprises: respectively acquiring a first metadata set maintained by the master node, a second metadata set maintained by the slave node, and a third metadata set maintained by the storage system, wherein the first metadata set records the storage state metadata information corresponding to each data cached on the master node, the second metadata set records the storage state metadata information corresponding to each data cached on the slave node, and the third metadata set records the storage state metadata information corresponding to each data stored on the storage system; by comparing the first metadata set, the second metadata set, and the third metadata set, determining the storage state of each data in the master node, the slave node, and the storage system, and screening the target data from the local cache data of the master node based on the storage state.

7. The method according to any one of claims 1 to 6, wherein, The method further comprises: In a case that the master node acquires a read request from a read end, the master node judges whether the to-be-read data meets a target data requirement according to storage state metadata information of the to-be-read data in response to the read request, wherein the read request is used to request reading of to-be-read data, and the target data requirement is used to indicate that the corresponding data exists in the master node cache storage module and is trusted in the master node cache storage module. In a case that the to-be-read data does not meet the target data requirement, the master node acquires the to-be-read data from a storage system, returns the to-be-read data in the storage system to the read end, and stores only the to-be-read data in the master node among the master-slave nodes.

8. The method according to any one of claims 1 to 7, wherein, The method further comprises: In a case that the master node acquires a write request from a write end, the master node writes to-be-written data corresponding to the write request and corresponding to-be-written metadata into a master node cache storage module, wherein the to-be-written metadata comprises storage state metadata information of the to-be-written data. The master node synchronizes the to-be-written data and the to-be-written metadata to the slave node, so that the slave node writes the to-be-written data and the to-be-written metadata into a slave node cache storage module; and the master node synchronizes the to-be-written data and the to-be-written metadata to the storage system for persistence by the storage system.

9. A data synchronization method of a cloud storage gateway, wherein, The cloud storage gateway comprises a master node and a slave node for providing a cloud storage gateway service to an object storage system; the method comprises: Determining a running state of the master-slave nodes for providing the cloud storage gateway service to the object storage system; In a case that the running state is switched from a master node independent running state to a master-slave replication running state, screening target data from local cache data of the master node, wherein the target data is neither successfully uploaded to the storage system nor successfully cached to the slave node; Starting a data cache synchronization task to synchronize the target data from the master node to the slave node.

10. The method of claim 9, wherein, The screening of the target data from the data cached in the master node comprises: Reading a metadata set maintained by the master node, wherein the metadata set records storage state metadata information corresponding to each data respectively, and the storage state metadata information is used to indicate whether the corresponding data is successfully uploaded to the object storage system and a node that has successfully cached the corresponding data; According to the storage state metadata information, the target data is determined.

11. The method of claim 10, wherein, The storage state metadata information corresponding to any data comprises: A first state variable, wherein when the first state variable is a first value, it indicates that the any data has been successfully uploaded to the object storage system, and when the first state variable is a second value, it indicates that the any data has not been successfully uploaded to the storage system; A second state variable, wherein a value of the second state variable is used to indicate a node that has successfully cached the any data.

12. An electronic device, comprising: Comprise: A processor; A memory for storing processor-executable instructions; A memory for storing processor-executable instructions; wherein the processor implements the method of any of claims 1-11 by executing the executable instructions.

13. A computer program product, wherein, comprising computer program / instructions to be executed by a processor to implement the steps of the method of any of claims 1-11.

14. A computer readable storage medium, wherein, having stored thereon computer instructions to be executed by a processor to implement the steps of the method of any of claims 1-11.

Images