Data synchronization method, apparatus, device, medium, and program product

Abstract

The disclosure provides a data synchronization method, which can be applied to the technical field of cloud computing, the technical field of distribution, the technical field of financial technology or other related fields. The method comprises the following steps: setting one of m parks as a master park for an access application, and setting other n parks as n backup parks for the access application; writing data of the access application into a service cluster of the master park to generate a first data file; synchronizing the data of the access application to service clusters of the n backup parks to generate n second data files; checking the consistency of each of the n second data files with the first data file; and in response to the inconsistency between the i-th second data file of the n second data files and the first data file, performing a resynchronization operation on the i-th second data file based on the first data file. The method ensures the consistency of the data of each park, improves the disaster recovery and backup capability of the platform, and enhances the security of the data.

Description

Technical Field

[0001] This disclosure relates to the fields of cloud computing technology and distributed technology, and more specifically to a data synchronization method, apparatus, device, medium and program product. Background Technology

[0002] Object storage service is a massive object-based storage service that provides customers with massive, secure, highly reliable, and low-cost data storage capabilities. Modern organizations need to create and analyze large amounts of unstructured data, such as photos, videos, emails, web pages, and audio files.

[0003] In object storage, disaster recovery is a crucial system metric. Disaster recovery is an abbreviation for disaster recovery and backup. Disaster recovery refers to establishing two or more functionally identical IT systems in the same city or different locations, allowing for health monitoring and function switching between them. If one location experiences an unexpected outage, the entire application system can be switched to the other location, ensuring continued operation. Backup, on the other hand, refers to users creating one or more copies of the data generated by the application to enhance data security.

[0004] Currently, most file systems employ dual-campus storage to enhance disaster recovery capabilities, meaning a complete data backup is stored in one campus. Due to the specific nature of the financial industry, storage platforms require 24 / 7 read / write services. Furthermore, while current platform services have data synchronization mechanisms between campuses during normal operating hours, they often cannot fully guarantee eventual data consistency across all centers. Summary of the Invention

[0005] In view of the above problems, according to a first aspect of this disclosure, embodiments of this disclosure provide a data synchronization method applied to an object storage platform, the object storage platform including service clusters deployed in m parks, where m is a positive integer greater than or equal to 2, the method comprising: responding to a request from an access application accessing the object storage platform, setting one of the m parks as the primary park for the access application, and setting the other n parks in the m parks as n backup parks for the access application, wherein n is a positive integer greater than or equal to 1 and less than or equal to m-1; responding to a write operation of the access application, writing the data of the access application into the service cluster of the primary park to generate a first data file; synchronizing the data of the access application to the service clusters of the n backup parks to generate n second data files; checking the consistency between each of the n second data files and the first data file through a file checking service; and responding to an inconsistency between the i-th second data file and the first data file among the n second data files, performing a resynchronization operation on the i-th second data file based on the first data file, wherein i is a positive integer greater than or equal to 1 and less than or equal to n.

[0006] According to some exemplary embodiments, the step of checking the consistency between each of the n second data files and the first data file through the file inspection service specifically includes: obtaining the size of the file block in the first data file and the size of the file block in the i-th second data file; comparing the size of the file block in the first data file with the size of the file block in the i-th second data file; and determining that the i-th second data file is inconsistent with the first data file in response to the inconsistency between the size of the file block in the first data file and the size of the file block in the i-th second data file.

[0007] According to some exemplary embodiments, obtaining the size of file blocks in the first data file and the size of file blocks in the i-th second data file specifically includes: continuously scanning closed file blocks in a first storage directory using a file inspection service process deployed on the service cluster of the main campus to obtain the size of file blocks in the first data file, wherein the first storage directory is a storage directory in the service cluster of the main campus used to store the data of the access application; and continuously scanning closed file blocks in a second storage directory using a file inspection service process deployed on the service cluster of the i-th backup campus to obtain the size of file blocks in the i-th second data file, wherein the second storage directory is a storage directory in the service cluster of the i-th backup campus used to store the data of the access application.

[0008] According to some exemplary embodiments, the step of checking the consistency between each of the n second data files and the first data file through the file inspection service further includes: in response to the fact that the size of the file block in the first data file is consistent with the size of the file block in the i-th second data file, obtaining the size of each object of the file block in the first data file and the size of each object of the file block in the i-th second data file; comparing the size of each object of the file block in the first data file with the size of each object of the file block in the i-th second data file respectively; and in response to the fact that the size of at least one object of the file block in the first data file is inconsistent with the size of at least one object of the file block in the i-th second data file, determining that the i-th second data file is inconsistent with the first data file.

[0009] According to some exemplary embodiments, the resynchronization operation on the i-th second data file based on the first data file includes: when the size of the j-th file block in the first data file is inconsistent with the size of the j-th file block in the i-th second data file, deleting the j-th file block in the i-th second data file and copying the j-th file block in the first data file to the storage location corresponding to the j-th file block in the i-th second data file.

[0010] According to some exemplary embodiments, after checking the consistency between each of the n second data files and the first data file, the method further includes: writing the file block information that was found to be inconsistent into a database; the resynchronization operation on the i-th second data file based on the first data file includes: periodically scanning the database by restoring the service to obtain the file block information that was found to be inconsistent; and performing the resynchronization operation during the idle period of the service cluster according to the obtained file block information.

[0011] According to some exemplary embodiments, the step of checking the consistency between each of the n second data files and the first data file through the file inspection service further includes: obtaining the size of the index file block in the first data file and the size of the index file block in the i-th second data file; comparing the size of the index file block in the first data file with the size of the index file block in the i-th second data file; and determining that the i-th second data file is inconsistent with the first data file in response to the inconsistency between the size of the index file block in the first data file and the size of the index file block in the i-th second data file.

[0012] According to some exemplary embodiments, the method further includes: in response to the data volume in the first data file being greater than a preset data volume threshold, starting multiple file check services and multiple recovery services, and performing the check and the resynchronization operation concurrently.

[0013] According to a second aspect of this disclosure, a data synchronization apparatus is also provided, applied to an object storage platform, the object storage platform comprising service clusters deployed in m parks, where m is a positive integer greater than or equal to 2, characterized in that the apparatus comprises:

[0014] The configuration module is used to: in response to the requirements of an access application accessing the object storage platform, set one of the m parks as the primary park for the access application, and set the other n parks among the m parks as n backup parks for the access application, where n is a positive integer greater than or equal to 1 and less than or equal to m-1;

[0015] The writing module is used to: respond to the write operation of the access application and write the data of the access application into the service cluster of the main campus to generate a first data file;

[0016] The synchronization module is used to: synchronize the data of the access application to the service clusters of the n backup parks to generate n second data files;

[0017] The file inspection module is used to: check the consistency between each of the n second data files and the first data file through a file inspection service; and

[0018] The resynchronization module is configured to: in response to the inconsistency between the i-th second data file and the first data file among the n second data files, perform a resynchronization operation on the i-th second data file based on the first data file, where i is a positive integer greater than or equal to 1 and less than or equal to n.

[0019] According to a third aspect of this disclosure, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0020] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided that stores executable instructions thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0021] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0022] The above one or more embodiments have the following advantages or beneficial effects: By synchronizing data across multiple centers using the method provided in this disclosure, the eventual consistency of data between each park (center) can be guaranteed, so that each center can take over business at any time, significantly improving the platform's disaster recovery and backup capabilities and enhancing data security. Attached Figure Description

[0023] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0024] Figure 1 The illustration schematically depicts application scenarios of data synchronization methods, apparatuses, devices, media, and program products according to embodiments of the present disclosure;

[0025] Figure 2 A flowchart illustrating a data synchronization method according to an embodiment of the present disclosure is shown schematically;

[0026] Figure 3 A schematic block diagram of a data synchronization apparatus according to an embodiment of the present disclosure is shown.

[0027] Figure 4 A block diagram of an electronic device for a data synchronization method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0028] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0029] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0030] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0031] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0032] In the technical solution disclosed herein, the acquisition, storage, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0033] First, the technical terms used in this article are explained and clarified as follows.

[0034] Kafka is a high-throughput distributed message queue system. It features a producer-consumer pattern, guarantees first-in-first-out (FIFO) ordering, does not lose data, and cleans up data every 7 days by default.

[0035] It should be noted that the data synchronization method and apparatus provided in this disclosure can be applied to the fields of cloud computing and distributed technology, as well as the financial field.

[0036] Figure 1 The illustration shows an application scenario diagram of a data synchronization method, apparatus, device, medium, and program product according to embodiments of the present disclosure.

[0037] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal devices 101, 102, and 103, network 104, and server 105. Network 104 is used as a medium to provide a communication link between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0038] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0039] Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0040] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0041] It should be noted that the privacy compliance detection method for IoT devices provided in this disclosure can generally be executed by server 105. Correspondingly, the privacy compliance detection device for IoT devices provided in this disclosure can generally be located in server 105. The privacy compliance detection method for IoT devices provided in this disclosure can also be executed by a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105. Correspondingly, the privacy compliance detection device for IoT devices provided in this disclosure can also be located in a server or server cluster that is different from server 105 and capable of communicating with terminal devices 101, 102, 103 and / or server 105.

[0042] It should be understood that Figure 2 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] Figure 2 A flowchart illustrating a data synchronization method according to an embodiment of the present disclosure is shown schematically.

[0044] Embodiments of this disclosure provide a data synchronization method, the flowchart of which can be found in [reference needed]. Figure 2 As shown, this specifically includes operations S1-S5.

[0045] In operation S1, in response to the request of the access application to access the object storage platform, one of the m parks is set as the primary park for the access application, and the other n parks in the m parks are set as n backup parks for the access application, where n is a positive integer greater than or equal to 1 and less than or equal to m-1.

[0046] In operation S2, in response to the write operation of the access application, the data of the access application is written into the service cluster of the main campus to generate a first data file.

[0047] In operation S3, the data of the access application is synchronized to the service clusters of the n backup parks to generate n second data files.

[0048] In operation S4, the consistency between each of the n second data files and the first data file is checked through the file inspection service.

[0049] In operation S5, in response to the inconsistency between the i-th second data file and the first data file among the n second data files, a resynchronization operation is performed on the i-th second data file based on the first data file, where i is a positive integer greater than or equal to 1 and less than or equal to n.

[0050] In the method provided in this disclosure, for an object storage platform with m zones, where m is a positive integer greater than or equal to 2, the service clusters in the m zones are first deployed, including storage services and Kafka cluster services. When an application accesses the platform, the primary zone information is set according to the application's requirements, and the remaining n zones are set as backup zones, where n is a positive integer greater than or equal to 1 and less than or equal to m-1. Setting a zone as the primary zone indicates that write operations by the application will only occur in the primary zone, generating a first data file. Then, the replicator process connects to the Kafka cluster, establishes multiple consumer groups, synchronizes the application's data to other backup zones, and generates a second data file in each backup zone. A file checking service then checks whether each of the n second data files is consistent with the first data file. If they are inconsistent, a resynchronization operation is performed on the i-th second data file. This method enables data synchronization across multiple centers, ensuring data consistency across the multi-center object storage platform, thus achieving high availability and high disaster recovery capabilities. Furthermore, the synchronized modification of inconsistent data files reduces subsequent manpower and material resources, saving costs. By leveraging the characteristics of clusters through a distributed system infrastructure and employing multi-replica storage for data written to access applications, the backup capabilities of the storage platform can be improved.

[0051] Furthermore, operation S4 specifically includes S41-S43.

[0052] In operation S41, the size of the file block in the first data file and the size of the file block in the i-th second data file are obtained.

[0053] In operation S42, the size of the file block in the first data file is compared with the size of the file block in the i-th second data file.

[0054] In operation S43, in response to the fact that the size of the file block in the first data file is inconsistent with the size of the file block in the i-th second data file, it is determined that the i-th second data file is inconsistent with the first data file.

[0055] In this embodiment, even after deploying the primary and backup zones and generating the corresponding data files, data inconsistencies may still occur. Therefore, based on the primary zone information set by the access application, it is assumed that the data in the primary zone is complete and accurate (i.e., if data inconsistencies occur, the data in the primary zone must be more than the data in the backup zone). In this case, the file-checker service is used to scan the HDFS file block sizes written by the user, using the data stored in the primary zone as a benchmark, to determine if they are consistent. When checking the consistency of data files using the file-checker service, the size of the file blocks in the data files is first obtained, then compared and determined, which improves the efficiency and accuracy of the check.

[0056] Furthermore, operation S41 specifically includes S411-S412.

[0057] In operation S411, the file inspection service process deployed on the service cluster of the main campus continuously scans the closed file blocks in the first storage directory to obtain the size of the file blocks in the first data file, wherein the first storage directory is the storage directory in the service cluster of the main campus used to store the data of the access application.

[0058] In operation S412, the file inspection service process deployed on the service cluster of the i-th backup zone continuously scans the closed file blocks in the second storage directory to obtain the size of the file blocks in the i-th second data file, wherein the second storage directory is the storage directory in the service cluster of the i-th backup zone used to store the data of the access application.

[0059] In the embodiments provided in this disclosure, by scanning closed file blocks in the storage directory, the size of the file blocks can be obtained relatively completely without affecting other related operations of the file inspection service.

[0060] In the embodiments provided in this disclosure, operation S4 further includes S44-S46.

[0061] In operation S44, in response to the fact that the size of the file block in the first data file is the same as the size of the file block in the i-th second data file, the size of each object of the file block in the first data file and the size of each object of the file block in the i-th second data file are obtained.

[0062] In operation S45, the size of each object in the file block of the first data file is compared with the size of each object in the file block of the i-th second data file.

[0063] In operation S46, in response to the fact that the size of at least one object of a file block in the first data file is inconsistent with the size of at least one object of a file block in the i-th second data file, it is determined that the i-th second data file is inconsistent with the first data file.

[0064] During comparisons, it's still possible that for a specific file block, the file size might be identical in both the primary and backup databases, but inconsistencies in user-stored objects could arise during file transfer. In such cases, the hdsf-object-checker service is needed to perform object checks on the databases where the primary and backup databases are inconsistent. Again, using the primary database as the benchmark, it checks whether each object stored in the primary database within that file block is consistent with each object stored in the backup database. Beyond comparing file block sizes, it further compares the sizes of individual objects within the file block. Compared to simply checking the file block size, this method is more efficient and accurate in detecting errors and also helps in subsequent error correction.

[0065] Furthermore, operation S5 specifically includes S51.

[0066] In operation S51, when the size of the j-th file block in the first data file is inconsistent with the size of the j-th file block in the i-th second data file, the j-th file block in the i-th second data file is deleted, and the j-th file block in the first data file is copied to the storage location corresponding to the j-th file block in the i-th second data file.

[0067] In the embodiments provided in this disclosure, if an inconsistency is found, the smaller file block in the backup zone is first deleted, and then the complete file block in the primary zone is copied to the backup zone. Similarly, HBase index file blocks can be copied from the primary zone to the backup zone. Modifying the file blocks in the backup zone during the comparison process ensures eventual consistency of data between the various zones (centers), allowing each center to take over business at any time, significantly improving the platform's disaster recovery and backup capabilities, and enhancing data security.

[0068] In the embodiments provided in this disclosure, after checking the consistency between each of the n second data files and the first data file, the method further includes: writing the information of the inconsistent file blocks into the database. Performing a resynchronization operation on the i-th second data file based on the first data file includes: periodically scanning the database by restoring the service to obtain the information of the inconsistent file blocks; and performing the resynchronization operation during the idle period of the service cluster based on the obtained file block information.

[0069] Inconsistent file block information is recorded in the database, and then the recovery service is used to retrieve the list of file block tasks to be repaired from the database. During idle periods, resynchronization operations are performed, which not only frees up some data storage space and reduces processing costs, but also improves synchronization efficiency by centralizing processing, making it more versatile and flexible.

[0070] Furthermore, operation S4 specifically includes S47-S49.

[0071] In operation S47, the size of the index file block in the first data file and the size of the index file block in the i-th second data file are obtained.

[0072] In operation S48, the size of the index file block in the first data file is compared with the size of the index file block in the i-th second data file.

[0073] In operation S49, in response to the fact that the size of the index file block in the first data file is inconsistent with the size of the index file block in the i-th second data file, it is determined that the i-th second data file is inconsistent with the first data file.

[0074] In the embodiments provided in this disclosure, the size of the index file blocks in the first data file and the i-th second data file is obtained and compared. Since the index file structure is a data structure built to accelerate search speed, it is frequently used in search engines, databases, and other fields in practical applications. Common index file structures include B+ trees, hash tables, and inverted indexes. Each of these structures has its own characteristics and application scenarios. In this embodiment, referencing the index file blocks can effectively improve search speed and efficiency.

[0075] In this embodiment of the disclosure, it further includes, in response to the data volume in the first data file being greater than a preset data volume threshold, starting multiple file check services and multiple recovery services, and performing the check and the resynchronization operation concurrently.

[0076] For data files that exceed the set data volume threshold, concurrent checks and resynchronization operations are performed. At the same time, a specific file check process can be specified to check file blocks in a specific directory. File checks are performed 24 hours a day on closed HDFS files (closed for more than 3 minutes), thereby ensuring the overall efficiency of the file check process.

[0077] Figure 3 A schematic block diagram of a data synchronization apparatus according to an embodiment of the present disclosure is shown.

[0078] like Figure 3As shown, the data synchronization device 300 according to this embodiment includes a setting module 310, a writing module 320, a synchronization module 330, a file checking module 340, and a resynchronization module 350.

[0079] The setting module 310 is configured to, in response to the request of an access application accessing the object storage platform, set one of the m parks as the primary park for the access application, and set the other n parks among the m parks as n backup parks for the access application, where n is a positive integer greater than or equal to 1 and less than or equal to m-1.

[0080] The writing module 320 is used to respond to the write operation of the access application and write the data of the access application into the service cluster of the main campus to generate a first data file.

[0081] The synchronization module 330 is used to synchronize the data of the access application to the service clusters of the n backup parks to generate n second data files.

[0082] The file inspection module 340 is used to check the consistency between each of the n second data files and the first data file through a file inspection service.

[0083] The resynchronization module 350 is used to perform a resynchronization operation on the i-th second data file based on the first data file in response to the inconsistency between the i-th second data file and the first data file among the n second data files, where i is a positive integer greater than or equal to 1 and less than or equal to n.

[0084] In embodiments of this disclosure, the file inspection module 340 is specifically configured to: obtain the size of a file block in the first data file and the size of a file block in the i-th second data file; compare the size of a file block in the first data file with the size of a file block in the i-th second data file; and, in response to a discrepancy between the size of a file block in the first data file and the size of a file block in the i-th second data file, determine that the i-th second data file is inconsistent with the first data file.

[0085] In embodiments of this disclosure, the file inspection module 340 can also be used to: continuously scan closed file blocks in a first storage directory using a file inspection service process deployed on the service cluster of the main campus to obtain the size of file blocks in the first data file, wherein the first storage directory is a storage directory in the service cluster of the main campus used to store the data of the access application; and continuously scan closed file blocks in a second storage directory using a file inspection service process deployed on the service cluster of the i-th backup campus to obtain the size of file blocks in the i-th second data file, wherein the second storage directory is a storage directory in the service cluster of the i-th backup campus used to store the data of the access application.

[0086] In embodiments of this disclosure, the file inspection module 340 may further be configured to: in response to the size of a file block in the first data file being consistent with the size of a file block in the i-th second data file, obtain the size of each object of a file block in the first data file and the size of each object of a file block in the i-th second data file; compare the size of each object of a file block in the first data file with the size of each object of a file block in the i-th second data file; and in response to the size of at least one object of a file block in the first data file being inconsistent with the size of at least one object of a file block in the i-th second data file, determine that the i-th second data file is inconsistent with the first data file.

[0087] In the embodiments of this disclosure, the resynchronization module 350 is specifically used to: when the size of the j-th file block in the first data file is inconsistent with the size of the j-th file block in the i-th second data file, delete the j-th file block in the i-th second data file and copy the j-th file block in the first data file to the storage location corresponding to the j-th file block in the i-th second data file.

[0088] In embodiments of this disclosure, the resynchronization module 350 can also be used to: write the detected inconsistent file block information into a database; the resynchronization operation on the i-th second data file based on the first data file includes: periodically scanning the database by restoring the service to obtain the detected inconsistent file block information; and performing the resynchronization operation during the idle period of the service cluster according to the obtained file block information.

[0089] In embodiments of this disclosure, the file inspection module 340 may further be used to: obtain the size of the index file block in the first data file and the size of the index file block in the i-th second data file; compare the size of the index file block in the first data file with the size of the index file block in the i-th second data file; and, in response to the inconsistency between the size of the index file block in the first data file and the size of the index file block in the i-th second data file, determine that the i-th second data file is inconsistent with the first data file.

[0090] In embodiments of this disclosure, the resynchronization module 350 can also be used to: in response to the data volume in the first data file being greater than a preset data volume threshold, initiate multiple file check services and multiple recovery services, and concurrently perform the check and the resynchronization operation.

[0091] Figure 4 A block diagram of an electronic device for a data synchronization method according to an embodiment of the present disclosure is shown schematically.

[0092] like Figure 4 As shown, an electronic device 400 according to an embodiment of the present invention includes a processor 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage portion 408 into a random access memory (RAM) 403. The processor 401 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 401 may also include onboard memory for caching purposes. The processor 401 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0093] RAM 403 stores various programs and data required for the operation of electronic device 400. Processor 401, ROM 402, and RAM 403 are interconnected via bus 404. Processor 401 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 402 and / or RAM 403. It should be noted that the programs may also be stored in one or more memories other than ROM 402 and RAM 403. Processor 401 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0094] According to an embodiment of the present invention, the electronic device 400 may further include an input / output (I / O) interface 405, which is also connected to a bus 404. The electronic device 400 may also include one or more of the following components connected to the I / O interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0095] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0096] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 402 and / or RAM 403 and / or one or more memories other than ROM 402 and RAM 403 described above.

[0097] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of the present invention.

[0098] When the computer program is executed by the processor 401, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0099] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via communication section 409, and / or installed from removable medium 411. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0100] In such an embodiment, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by processor 401, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0101] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

Claims

1. A data synchronization method applied to an object storage platform, the object storage platform comprising service clusters deployed in m parks, where m is a positive integer greater than or equal to 2, characterized in that, The method includes: In response to the requirements of the access application accessing the object storage platform, one of the m parks is set as the primary park for the access application, and the other n parks among the m parks are set as n backup parks for the access application, where n is a positive integer greater than or equal to 1 and less than or equal to m-1. In response to the write operation of the access application, the data of the access application is written into the service cluster of the main campus to generate a first data file; The data from the access application is synchronized to the service clusters of the n backup parks to generate n second data files; The file inspection service checks the consistency between each of the n second data files and the first data file, including: obtaining the size of file blocks in the first data file and the size of file blocks in the i-th second data file; comparing the size of file blocks in the first data file with the size of file blocks in the i-th second data file; and determining that the i-th second data file is inconsistent with the first data file in response to a discrepancy between the size of file blocks in the first data file and the size of file blocks in the i-th second data file. Specifically, the file inspection service process deployed on the service cluster of the main campus continuously scans closed file blocks in the first storage directory to obtain the size of file blocks in the first data file, where the first storage directory is the storage directory in the service cluster of the main campus used to store the data of the access application; and the file inspection service process deployed on the service cluster of the i-th backup campus continuously scans closed file blocks in the second storage directory to obtain the size of file blocks in the i-th second data file, where the second storage directory is the storage directory in the service cluster of the i-th backup campus used to store the data of the access application. In response to the inconsistency between the i-th second data file and the first data file among the n second data files, a resynchronization operation is performed on the i-th second data file based on the first data file, where i is a positive integer greater than or equal to 1 and less than or equal to n.

2. The method according to any one of claims 1, characterized in that, The step of checking the consistency between each of the n second data files and the first data file through the file inspection service further includes: In response to the fact that the size of the file block in the first data file is the same as the size of the file block in the i-th second data file, the size of each object in the file block in the first data file and the size of each object in the file block in the i-th second data file are obtained. Compare the size of each object in the file block of the first data file with the size of each object in the file block of the i-th second data file; and In response to the fact that the size of at least one object of a file block in the first data file is inconsistent with the size of at least one object of a file block in the i-th second data file, it is determined that the i-th second data file is inconsistent with the first data file.

3. The method according to claim 2, characterized in that, The resynchronization operation on the i-th second data file based on the first data file includes: When the size of the j-th file block in the first data file is inconsistent with the size of the j-th file block in the ith second data file, delete the j-th file block in the ith second data file and copy the j-th file block in the first data file to the storage location corresponding to the j-th file block in the ith second data file.

4. The method according to any one of claims 1 and 3, characterized in that, After checking the consistency between each of the n second data files and the first data file, the method further includes: writing the file block information that was found to be inconsistent into a database; The resynchronization operation on the i-th second data file based on the first data file includes: periodically scanning the database by restoring the service to obtain information on inconsistent file blocks; and performing the resynchronization operation during the idle period of the service cluster according to the obtained file block information.

5. The method according to claim 1, characterized in that, The step of checking the consistency between each of the n second data files and the first data file through the file inspection service further includes: Obtain the size of the index file block in the first data file and the size of the index file block in the i-th second data file; Compare the size of the index file block in the first data file with the size of the index file block in the i-th second data file; and In response to the fact that the size of the index file block in the first data file is inconsistent with the size of the index file block in the i-th second data file, it is determined that the i-th second data file is inconsistent with the first data file.

6. The method according to claim 4, characterized in that, The method further includes: in response to the data volume in the first data file being greater than a preset data volume threshold, starting multiple file check services and multiple recovery services, and performing the check and the resynchronization operation concurrently.

7. A data synchronization device applied to an object storage platform, the object storage platform comprising service clusters deployed in m parks, where m is a positive integer greater than or equal to 2, characterized in that, The device includes: The configuration module is used to: in response to the requirements of an access application accessing the object storage platform, set one of the m parks as the primary park for the access application, and set the other n parks among the m parks as n backup parks for the access application, where n is a positive integer greater than or equal to 1 and less than or equal to m-1; The writing module is used to: respond to the write operation of the access application and write the data of the access application into the service cluster of the main campus to generate a first data file; The synchronization module is used to: synchronize the data of the access application to the service clusters of the n backup parks to generate n second data files; The file inspection module is used to: check the consistency between each of the n second data files and the first data file through a file inspection service, including: obtaining the size of file blocks in the first data file and the size of file blocks in the i-th second data file; comparing the size of file blocks in the first data file with the size of file blocks in the i-th second data file; and determining that the i-th second data file is inconsistent with the first data file in response to the inconsistency between the size of file blocks in the first data file and the size of file blocks in the i-th second data file; wherein, the file inspection service process deployed on the service cluster of the main campus continuously scans the closed file blocks in the first storage directory to obtain the size of file blocks in the first data file, wherein the first storage directory is the storage directory in the service cluster of the main campus used to store the data of the access application; and the file inspection service process deployed on the service cluster of the i-th backup campus continuously scans the closed file blocks in the second storage directory to obtain the size of file blocks in the i-th second data file, wherein the second storage directory is the storage directory in the service cluster of the i-th backup campus used to store the data of the access application; and The resynchronization module is configured to: in response to the inconsistency between the i-th second data file and the first data file among the n second data files, perform a resynchronization operation on the i-th second data file based on the first data file, where i is a positive integer greater than or equal to 1 and less than or equal to n.

8. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images