Data synchronization method, system, electronic device and storage medium

Abstract

The present invention relates to the field of computer technology, and discloses a data synchronization method, system, electronic device and storage medium, including: creating a shared memory at a main site to intercept initial synchronization data sent by several cloud hosts on the main site based on the shared memory; caching the initial synchronization data to a preset cache storage pool, and recording index information of the initial synchronization data; generating a bitmap file of the initial synchronization data according to the index information of the initial synchronization data when the cache situation of the preset cache storage pool meets the preset synchronization conditions; generating target synchronization data according to the bitmap file, and synchronizing the target synchronization data to a backup site. The method provided by the above scheme generates a bitmap file of the initial synchronization data, and generates target synchronization data according to the bitmap file. Subsequently, only the target synchronization data needs to be synchronized to the backup site, which reduces the occupation of cloud host resources by data synchronization, thereby improving the performance of the cloud host.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data synchronization method, system, electronic device and storage medium. Background Art

[0002] With the development of cloud computing, the scale of cloud data centers continues to increase, and the requirements for data security are becoming higher and higher. Dual-site disaster recovery systems have emerged. In dual-site disaster recovery systems, the primary site synchronizes business data to the backup site so that the backup site can take over the business when the primary site fails. Therefore, in order to ensure the reliability of disaster recovery, how to synchronize data between the primary site and the backup site has become a key research topic.

[0003] In the related art, the cloud host deployed on the primary site usually synchronizes the business data to the backup site in real time every time it generates business data. However, the real-time transmission of business data will occupy the cloud host resources, which will cause a significant decline in the performance of the cloud host in scenarios with frequent business interactions. Summary of the invention

[0004] The present application provides a data synchronization method, system, electronic device and storage medium to solve the defects of related technologies such as reducing the performance of the cloud host on the primary site.

[0005] The first aspect of the present application provides a data synchronization method, comprising:

[0006] Creating a shared memory at the primary site to intercept initial synchronization data sent by several cloud hosts at the primary site based on the shared memory;

[0007] Cache the initial synchronization data to a preset cache storage pool, and record index information of the initial synchronization data;

[0008] When the cache condition of the preset cache storage pool meets the preset synchronization condition, generating a bitmap file of the initial synchronization data according to the index information of the initial synchronization data;

[0009] According to the bitmap file, target synchronization data is generated, and the target synchronization data is synchronized to the backup site; wherein the data volume of the target synchronization data is smaller than the data volume of the initial synchronization data.

[0010] In an optional embodiment, the method further includes:

[0011] According to a preset identification cycle, a data consistency identification is issued to the shared memory, and the data consistency identification is added to the tail of the initial synchronization data intercepted at the corresponding time;

[0012] In the process of caching the initial synchronization data to a preset cache storage pool, when the data consistency identifier carried at the end of the initial synchronization data is identified, the cache disk is split in the preset cache storage pool to replace the cache disk to store subsequent initial synchronization data, so that the initial synchronization data cached by the same cache disk belongs to the same preset identifier period.

[0013] In an optional embodiment, the method further includes:

[0014] When the cache disk is replaced, it is determined that the cache status of the preset cache storage pool meets the preset synchronization condition, so as to synchronize the cached data of the previous cache disk to the backup site.

[0015] In an optional implementation, caching the initial synchronization data into a preset cache storage pool includes:

[0016] Obtaining a write fixed unit length of the preset cache storage pool;

[0017] The initial synchronization data is cached in a preset cache storage pool according to the write fixed unit length.

[0018] In an optional implementation, the caching the initial synchronization data to a preset cache storage pool according to the write fixed unit length includes:

[0019] According to the write fixed unit length, the initial synchronization data is divided and spliced ​​to record the initial synchronization data as a plurality of data to be written; wherein the length of each piece of the data to be written is equal to the write fixed unit length;

[0020] The plurality of to-be-written data are cached in a preset cache storage pool.

[0021] In an optional implementation, generating a bitmap file of the initial synchronization data according to the index information of the initial synchronization data includes:

[0022] Determine the virtual disk offset and data length of each piece of the initial synchronization data according to the index information of the initial synchronization data;

[0023] According to the sending order of each piece of the initial synchronization data and based on the virtual disk offset and data length of each piece of the initial synchronization data, a bitmap file of the initial synchronization data is generated.

[0024] In an optional implementation, generating target synchronization data according to the bitmap file includes:

[0025] According to the bitmap file, all the initial synchronization data in the previous cache disk are deduplicated and merged to obtain target synchronization data.

[0026] In an optional implementation, the recording of index information of the initial synchronization data includes:

[0027] Obtaining a virtual disk offset, a data length, and a cache disk offset of each piece of the initial synchronization data;

[0028] Recording index information of the initial synchronization data according to the virtual disk offset, data length and cache disk offset of each piece of the initial synchronization data;

[0029] The initial synchronization data is sent from the virtual disk to the shared memory by the cloud host, and the initial synchronization data is cached in the cache disk of the preset cache storage pool.

[0030] In an optional embodiment, the method further includes:

[0031] Before the cloud host on the primary site sends the initial synchronization data, a freeze instruction is sent to the cloud host, so that the cloud host will not modify the data during the process of sending the initial synchronization data.

[0032] In an optional embodiment, the method further includes:

[0033] According to the preset application consistency logic, an application consistency identifier is issued to the shared memory, and the application consistency identifier is added to the tail of the initial synchronization data intercepted at the corresponding time;

[0034] In the process of caching the initial synchronization data to a preset cache storage pool, when the application consistency identifier carried at the end of the initial synchronization data is identified, the cache disk is split in the preset cache storage pool to replace the cache disk to store subsequent initial synchronization data, so that the initial synchronization data cached by the same cache disk belongs to the same application.

[0035] In an optional embodiment, the method further includes:

[0036] The recovery point of the target synchronization data is determined according to the sending time of the application consistency identifier represented by the cache disk segmentation result of the preset cache storage pool, so that the backup site performs service recovery on the target synchronization data according to the recovery point.

[0037] A second aspect of the present application provides a data synchronization device, including:

[0038] A creation module, used to create a shared memory at the primary site, so as to intercept initial synchronization data sent by several cloud hosts at the primary site based on the shared memory;

[0039] A cache module, used to cache the initial synchronization data into a preset cache storage pool and record index information of the initial synchronization data;

[0040] A generating module, configured to generate a bitmap file of the initial synchronization data according to the index information of the initial synchronization data when the cache condition of the preset cache storage pool meets the preset synchronization condition;

[0041] A synchronization module is used to generate target synchronization data according to the bitmap file, and synchronize the target synchronization data to the backup site; wherein the data volume of the target synchronization data is smaller than the data volume of the initial synchronization data.

[0042] A third aspect of the present application provides a data synchronization system, comprising: a primary site and a backup site, wherein the primary site is deployed with a plurality of cloud hosts;

[0043] The primary site synchronizes the initial synchronization data sent by the cloud host to the backup site based on the data synchronization method described in the first aspect and various possible designs of the first aspect.

[0044] A fourth aspect of the present application provides an electronic device, comprising: at least one processor and a memory;

[0045] The memory stores computer-executable instructions;

[0046] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method described in the first aspect and various possible designs of the first aspect.

[0047] A fifth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When a processor executes the computer-executable instructions, the method described in the first aspect and various possible designs of the first aspect are implemented.

[0048] The sixth aspect of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the method described in the first aspect and various possible designs of the first aspect.

[0049] The technical solution of this application has the following advantages:

[0050] The present application provides a data synchronization method, system, electronic device and storage medium, the method comprising: creating a shared memory at the main site to intercept the initial synchronization data sent by several cloud hosts on the main site based on the shared memory; caching the initial synchronization data to a preset cache storage pool, and recording the index information of the initial synchronization data; generating a bitmap file of the initial synchronization data according to the index information of the initial synchronization data when the cache status of the preset cache storage pool meets the preset synchronization conditions; generating target synchronization data according to the bitmap file, and synchronizing the target synchronization data to a backup site; wherein the data volume of the target synchronization data is less than the data volume of the initial synchronization data. The method provided by the above scheme generates a bitmap file of the initial synchronization data, and generates the target synchronization data according to the bitmap file. Subsequently, only the target synchronization data needs to be synchronized to the backup site, which reduces the occupation of cloud host resources by data synchronization, thereby improving the performance of the cloud host. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technical descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0052] Figure 1 A flowchart of a data synchronization method provided in an embodiment of the present application;

[0053] Figure 2 A schematic diagram of an application architecture of a data synchronization method provided in an embodiment of the present application;

[0054] Figure 3 A schematic diagram of the data cache structure of the cache disk provided in an embodiment of the present application;

[0055] Figure 4 A schematic diagram of the data structure of a bitmap file provided in an embodiment of the present application;

[0056] Figure 5 A schematic diagram of an application architecture of another data synchronization method provided in an embodiment of the present application;

[0057] Figure 6 A schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application;

[0058] Figure 7 A schematic diagram of the structure of a data synchronization system provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0060] The above drawings have shown clear embodiments of the present application, which will be described in more detail below. These drawings and text descriptions are not intended to limit the scope of the present disclosure in any way, but to illustrate the concepts of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] In addition, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. In the description of the following embodiments, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0063] In related technologies, with the development of cloud computing, the scale of cloud data centers continues to increase, and the requirements for data security are becoming higher and higher. Dual-site disaster recovery systems have emerged. The key idea of ​​dual-site disaster recovery is to deploy and maintain two completely independent data centers or sites in two different geographical locations, called the primary site and the backup site. The primary site is the main operating location, which is responsible for processing normal business traffic and storing data. The backup site is a completely independent site, whose main purpose is to take over business traffic and data processing when the primary site fails or is unavailable.

[0064] In the increasingly complex application scenarios and business scenarios with higher performance requirements, higher requirements are placed on the disaster recovery system without affecting the business continuity and performance of the primary site. In order to meet the needs of dual-site disaster recovery, the data in the virtual machine (cloud host) needs to be transmitted in real time. However, in actual use, due to the frequent data interaction between the virtual machine and the physical machine during the operation of the virtual machine, if all data is transmitted in real time, the performance of the virtual machine will be significantly reduced, affecting the user experience. If real-time transmission is not possible, business continuity cannot be maintained and the needs of dual-site disaster recovery cannot be met.

[0065] In response to the above problems, an embodiment of the present application provides a data synchronization method, system, electronic device and storage medium, the method comprising: creating a shared memory at the main site to intercept the initial synchronization data sent by several cloud hosts on the main site based on the shared memory; caching the initial synchronization data to a preset cache storage pool, and recording the index information of the initial synchronization data; generating a bitmap file of the initial synchronization data according to the index information of the initial synchronization data when the cache status of the preset cache storage pool meets the preset synchronization conditions; generating target synchronization data according to the bitmap file, and synchronizing the target synchronization data to the backup site; wherein the data volume of the target synchronization data is less than the data volume of the initial synchronization data. The method provided by the above scheme generates a bitmap file of the initial synchronization data, and generates the target synchronization data according to the bitmap file. Subsequently, only the target synchronization data needs to be synchronized to the backup site, which reduces the occupation of cloud host resources by data synchronization, thereby improving the performance of the cloud host.

[0066] The following specific embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0067] The embodiment of the present application provides a data synchronization method for synchronizing data between a primary site and a backup site. The execution subject of the embodiment of the present application is an electronic device, such as a server, a desktop computer, a laptop computer, a tablet computer, and other electronic devices that can be used for data synchronization.

[0068] like Figure 1 FIG. 1 is a flow chart of a data synchronization method provided in an embodiment of the present application, and the method includes:

[0069] Step 101 : Create a shared memory at the primary site to intercept initial synchronization data sent by several cloud hosts at the primary site based on the shared memory.

[0070] It should be noted that each primary site can be deployed with multiple cloud hosts, and the cloud host sends the I / O data on the virtual disk as initial synchronization data to the primary site, so as to synchronize the I / O data to the backup site based on the primary site.

[0071] Specifically, a management process (CDP-WORK task) for each virtual disk may be created in advance, so as to create a shared memory based on the management process, and then intercept the initial synchronization data sent by the cloud host (virtual machine) based on the shared memory.

[0072] Step 102: cache the initial synchronization data to a preset cache storage pool, and record index information of the initial synchronization data.

[0073] Specifically, the initial synchronization data can be written to the cache disk of the preset cache storage pool in the order of interception of the initial synchronization data, so that the initial synchronization data can be cached to the preset cache storage pool, and index information such as the virtual disk offset, data length and cache disk offset of the initial synchronization data can be recorded.

[0074] Specifically, in one embodiment, the virtual disk offset, data length and cache disk offset of each initial synchronization data can be obtained; based on the virtual disk offset, data length and cache disk offset of each initial synchronization data, the index information of the initial synchronization data is recorded.

[0075] The initial synchronization data is sent from the virtual disk to the shared memory by the cloud host, and the initial synchronization data is cached in the cache disk of the preset cache storage pool.

[0076] Step 103: When the cache status of the preset cache storage pool meets the preset synchronization condition, a bitmap file of the initial synchronization data is generated according to the index information of the initial synchronization data.

[0077] Specifically, when the cache status of the preset cache storage pool meets the preset synchronization condition, the data to be synchronized can be located in all the obtained initial synchronization data, and then the bitmap files of the data to be synchronized can be generated.

[0078] Step 104: Generate target synchronization data according to the bitmap file, and synchronize the target synchronization data to the backup site.

[0079] The data volume of the target synchronization data is smaller than the data volume of the initial synchronization data.

[0080] Specifically, the data to be synchronized including multiple pieces of initial synchronization data can be deduplicated and merged according to the bitmap file to obtain target synchronization data, so as to reduce the amount of data synchronization and reduce the occupation of cloud host resources by data synchronization, thereby improving the performance of the cloud host.

[0081] Specifically, before synchronizing the target synchronization data to the backup site, the target synchronization data may be subjected to data content analysis to identify redundant content and repeated content, obtain content analysis results of the target synchronization data, and then apply a compression algorithm to encode the target synchronization data according to the content analysis results to obtain compressed target synchronization data, and finally send the compressed target synchronization data to the backup site. After obtaining the compressed target synchronization data, the backup site uses a corresponding decompression algorithm to restore the original data to obtain the target synchronization data.

[0082] On the basis of the above embodiment, there is a main site with a virtual machine containing multiple virtual disks, one virtual disk corresponds to one CDP-WORK process, the IO data volume of each process is different and the IO issuance time is also inconsistent, so it is impossible to accurately guarantee the time consistency of multi-virtual disk IO interception. If multiple cache disks are at the same time, the data consistency cannot be guaranteed, where the cache disk corresponds to the virtual disk one by one. After a disaster occurs, due to factors such as the file system, there will be obstacles to recovering the data, and even recovery failure will occur, resulting in data loss.

[0083] In order to solve the above technical problem, as an implementable manner, based on the above embodiment, in one embodiment, the method further includes:

[0084] Step 201: Send a data consistency identifier to the shared memory according to a preset identifier cycle, and add the data consistency identifier to the tail of the initial synchronization data intercepted at the corresponding time;

[0085] Step 202, in the process of caching the initial synchronization data to the preset cache storage pool, when the data consistency identifier carried at the end of the initial synchronization data is identified, the cache disk is split in the preset cache storage pool to replace the cache disk to store subsequent initial synchronization data, so that the initial synchronization data cached by the same cache disk belongs to the same preset identification period.

[0086] Among them, Figure 2 As shown, it is a schematic diagram of the application architecture of a data synchronization method provided in an embodiment of the present application. Specifically, an identification transmission interface (QMP interface) can be added to the cloud host, and scheduling can be performed through the upper-layer business. A sync identification (data consistency identification) can be forcibly added at the intercepted I / O. That is, the identification is added to multiple shared memories at the same time. After each management process recognizes the identification, the current incremental disk (cache disk) is cut to generate a new cache disk, and subsequent data falls into the new cache disk. At this time, it can be guaranteed that the previous I / O data is the data sent by the virtual machine in the previous preset identification cycle, thereby ensuring the consistency of multi-disk data.

[0087] Specifically, the data consistency identifier can be injected into all shared memories on the cloud host side through the identifier transmission interface. If any data is currently being intercepted, the identifier is added to the end of the data. The recognition function of the data consistency identifier is added to the management process. When the data consistency identifier is identified, after ensuring that the IO data before the identifier has been saved in the cache disk, a new cache disk is established to store subsequent IO data, so that the initial synchronization data cached by the same cache disk belongs to the same preset identification period.

[0088] The data consistency mark is only used to split the cache disk, and no data is saved. The preset mark period should be greater than or equal to the original cache disk creation period to avoid frequent cache disk splitting.

[0089] Specifically, in one embodiment, when the cache disk is replaced, it is determined that the cache status of the preset cache storage pool meets the preset synchronization condition, so as to synchronize the cached data of the previous cache disk to the backup site.

[0090] Specifically, when the subsequently intercepted original synchronization data is cached to the newly split cache disk, no new data will be written to the previous cache disk, so the cached data of the previous cache disk can be synchronized to the backup site to release the cache disk resources.

[0091] On the basis of the above embodiments, in the cloud host file system, read and write operations are usually performed in fixed units, such as 4K-byte read and write units. When the data to be written is smaller than the size of this unit, in order to complete the write operation, the system needs to first read the contents of the entire unit, modify the data, and then write the entire unit back, that is, even if the actual written data is less than 4K bytes, 4K bytes of data need to be written. This process generates write amplification. Since the management process creates a cache disk according to the set identification cycle and caches the actual size of the initial synchronization data, a write amplification problem exists on the basis of the file system, resulting in the degradation of the disaster recovery function.

[0092] In order to solve the above technical problem, as an implementable method, based on the above embodiment, in one embodiment, initial synchronization data is cached to a preset cache storage pool, including:

[0093] Step 1021, obtaining a write fixed unit length of a preset cache storage pool;

[0094] Step 1022: Cache the initial synchronization data into a preset cache storage pool according to the write fixed unit length.

[0095] The fixed write unit length is the length of the fixed read and write operation unit. For example, the fixed write unit length of the preset cache storage pool is 1M.

[0096] Specifically, in one embodiment, the initial synchronization data can be divided and spliced ​​according to the write fixed unit length to record the initial synchronization data as a number of data to be written; wherein the length of each data to be written is equal to the write fixed unit length; and the several data to be written are cached in a preset cache storage pool.

[0097] Specifically, Figure 3As shown, it is a schematic diagram of the data cache structure of the cache disk provided in an embodiment of the present application. When the initial synchronization data 1, initial synchronization data 2, initial synchronization data 3 and initial synchronization data 4 (referred to as I / O1~4) are intercepted in sequence, the initial synchronization data is divided and spliced ​​according to the write fixed unit length. If the sum of the lengths of the initial synchronization data 1, initial synchronization data 2 and initial synchronization data 3 is 2 write fixed unit lengths, and the lengths of the initial synchronization data 1, initial synchronization data 2 and initial synchronization data 3 are all less than the write fixed unit length, the initial synchronization data 2 is divided, and the divided fragments are spliced ​​to the initial synchronization data 1 and the initial synchronization data 3 to obtain 2 data to be written, and the 2 data to be written are cached in the preset cache storage pool in sequence, that is, when the cumulative intercepted data length exceeds the write fixed unit length, a write operation is performed to complete the cache of a piece of data to be written. Among them, the index information of the initial synchronization data is cached in the index storage area of ​​the cache disk in the interception order of the initial synchronization data in sequence to realize the associated cache of the index information and the initial synchronization data.

[0098] Specifically, in one embodiment, the virtual disk offset and data length of each piece of initial synchronization data can be determined according to the index information of the initial synchronization data; a bitmap file of the initial synchronization data is generated according to the virtual disk offset and data length of each piece of initial synchronization data in the order in which the initial synchronization data are sent.

[0099] The order in which the initial synchronization data is sent is the order in which the initial synchronization data is intercepted.

[0100] Furthermore, in one embodiment, all initial synchronization data in the previous cache disk may be deduplicated and merged according to the bitmap file to obtain target synchronization data.

[0101] Specifically, if the initial synchronization data cached in the previous cache disk is the data to be synchronized, all the initial synchronization data in the previous cache disk are deduplicated and merged according to the bitmap file of the previous cache disk to obtain the target synchronization data.

[0102] For example, Figure 4As shown, it is a schematic diagram of the data structure of the bitmap file provided in the embodiment of the present application. The final format of the bitmap content displayed to the outside is: {offset,length} = {cache_offset,length}+…+ {cache_offset,length}, where cache_offset represents the cache disk offset of the initial synchronization data, and length represents the data length of the initial synchronization data. Assuming that the cache disk currently caches initial synchronization data 1 to 9 (abbreviated as io[1] to io[9]), a bitmap file of the initial synchronization data can be generated according to the cache disk offset and data length corresponding to each initial synchronization data. Figure 4 The position of the bitmap module corresponding to each initial synchronization data identifies the starting position of the initial synchronization data in the disk address space. The length of the bitmap module indicates the data length of the initial synchronization data. Finally, the initial synchronization data 1 to 9 are deduplicated according to the position and length of each bitmap module to obtain the data to be merged (io_i) corresponding to each initial synchronization data. Finally, these data to be merged are merged to obtain the target synchronization data.

[0103] On the basis of the above embodiments, in order to ensure that key applications resume operation within the allowed time range, minimize the losses caused by disasters, and make users basically unaware of the occurrence of disasters, it is necessary to make the services provided by the system complete, reliable and secure. Therefore, it is necessary to add application consistency processing on the basis of multi-disk data consistency to ensure that the file system data is completely sent to the disk to ensure that it is completely intercepted. As an implementable manner, on the basis of the above embodiments, in one embodiment, the method further includes:

[0104] Step 301, before the cloud host on the primary site sends the initial synchronization data, a freeze instruction is sent to the cloud host, so that the cloud host will not modify the data in the process of sending the initial synchronization data.

[0105] It should be noted that the freeze instruction is used to trigger the freeze operation, which is used to freeze the state of the file system during a specific operation, ensuring that all data in the file system remains consistent during the operation, ensuring the stability of the file system during consistency checks or repairs, and preventing data from being modified during the checks or repairs. All write operations to the file system will be paused or delayed, and will not prevent read operations on the file system. Read operations can continue to ensure that users and applications can access data.

[0106] Specifically, before the cloud host on the primary site sends the initial synchronization data, a freeze instruction is sent to the cloud host to prevent the cloud host from modifying the initial synchronization data being sent during the process of sending the initial synchronization data, resulting in inconsistency between the sent data and the actual data of the cloud host.

[0107] Specifically, in one embodiment, an application consistency identifier can be sent to the shared memory according to the preset application consistency logic, and the application consistency identifier can be added to the tail of the initial synchronization data intercepted at the corresponding moment; in the process of caching the initial synchronization data to the preset cache storage pool, when the application consistency identifier carried at the tail of the initial synchronization data is identified, the cache disk is split in the preset cache storage pool to replace the cache disk to store subsequent initial synchronization data, so that the initial synchronization data cached by the same cache disk belongs to the same application.

[0108] Specifically, Figure 5 As shown, it is a schematic diagram of the application architecture of another data synchronization method provided in an embodiment of the present application. The upper-layer business registers a freeze instruction in the cloud host according to the customer configuration cycle represented by the preset application consistency logic, and then executes the flush action to allow the I / O data cached by the file system to fall into the cache disk, thereby being intercepted, and then adding an Fsync identifier (application consistency identifier) ​​to each cache disk.

[0109] Specifically, by adding a freeze operation, the application consistency of the synchronized data is guaranteed to avoid the situation where the data subsequently synchronized to the backup site cannot run the corresponding application normally. Specifically, the upper layer can send a freeze operation to the cloud host through IVA to prohibit I / O write operations, and then perform a flush operation to flush the file system cache to the disk; finally, according to the logic of multi-disk data application consistency, the application consistency identifier is issued, and when the application consistency identifier carried at the end of the initial synchronization data is identified, the cache disk is split in the preset cache storage pool to replace the cache disk to store the subsequent initial synchronization data, so that the initial synchronization data cached by the same cache disk belongs to the same application.

[0110] Accordingly, in one embodiment, the recovery point of the target synchronization data can be determined according to the delivery time of the application consistency identifier represented by the cache disk segmentation result of the preset cache storage pool, so that the backup site can perform business recovery on the target synchronization data according to the recovery point.

[0111] It should be noted that the application consistency identifier is not only used for the consistency of multi-disk data, but also used to indicate that the previous data is a reliable recovery point and the business of the current node at the backup site is available.

[0112] Specifically, the backup site can create a recovery point for the target synchronization data based on the time of issuing the application consistency identifier represented by the cache disk segmentation result. The recovery point is actually the time point of recovery, so that the backup site can restore the business application practiced at the corresponding time based on the target synchronization data.

[0113] The data synchronization method provided in the embodiment of the present application is to create a shared memory at the main site to intercept the initial synchronization data sent by several cloud hosts on the main site based on the shared memory; cache the initial synchronization data to a preset cache storage pool, and record the index information of the initial synchronization data; when the cache situation of the preset cache storage pool meets the preset synchronization conditions, generate a bitmap file of the initial synchronization data according to the index information of the initial synchronization data; generate target synchronization data according to the bitmap file, and synchronize the target synchronization data to the backup site; wherein the data volume of the target synchronization data is less than the data volume of the initial synchronization data. The method provided by the above scheme generates a bitmap file of the initial synchronization data, and generates the target synchronization data according to the bitmap file. Subsequently, only the target synchronization data needs to be synchronized to the backup site, which reduces the occupation of cloud host resources by data synchronization, thereby improving the performance of the cloud host. In addition, the multi-disk data consistency and application consistency functions are realized, and the impact of the write amplification problem of the file system on disaster recovery performance is solved.

[0114] An embodiment of the present application provides a data synchronization device for executing the data synchronization method provided in the above embodiment.

[0115] like Figure 6 , which is a schematic diagram of the structure of a data synchronization device provided in an embodiment of the present application. The data synchronization device 60 comprises: a creation module 601 , a cache module 602 , a generation module 603 and a synchronization module 604 .

[0116] Among them, a creation module is used to create a shared memory at the main site to intercept the initial synchronization data sent by several cloud hosts on the main site based on the shared memory; a cache module is used to cache the initial synchronization data to a preset cache storage pool and record the index information of the initial synchronization data; a generation module is used to generate a bitmap file of the initial synchronization data according to the index information of the initial synchronization data when the cache situation of the preset cache storage pool meets the preset synchronization conditions; a synchronization module is used to generate target synchronization data according to the bitmap file, and synchronize the target synchronization data to the backup site; wherein the data volume of the target synchronization data is less than the data volume of the initial synchronization data.

[0117] Regarding the data synchronization device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0118] The data synchronization device provided in the embodiment of the present application is used to execute the data synchronization method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0119] An embodiment of the present application provides a data synchronization system for executing the data synchronization method provided in the above embodiment.

[0120] like Figure 7 , which is a schematic diagram of the structure of a data synchronization system provided in an embodiment of the present application, the system includes: a primary site and a backup site, and the primary site is deployed with several cloud hosts.

[0121] The primary site synchronizes the initial synchronization data sent by the cloud host to the backup site based on the data synchronization method provided in the above embodiment.

[0122] Specifically, each virtual disk under each cloud host corresponds to a management process. The management process creates a shared memory to intercept the initial synchronization data of the corresponding disk sent by the virtual machine. The management process obtains data from the shared memory and records it in the cache storage pool. The recording method consists of a bitmap and an incremental disk (cache disk). The bitmap and incremental disk are sent to the backup site periodically according to the set period. The backup site merges the data to complete the disaster recovery function.

[0123] The data synchronization system provided in the embodiment of the present application is used to execute the data synchronization method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0124] An embodiment of the present application provides an electronic device for executing the data synchronization method provided in the above embodiment.

[0125] like Figure 8 FIG. 8 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 80 includes: at least one processor 81 and a memory 82 .

[0126] The memory stores computer-executable instructions; at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the data synchronization method provided in the above embodiment.

[0127] The electronic device provided in the embodiment of the present application is used to execute the data synchronization method provided in the above embodiment. Its implementation method and principle are the same and will not be repeated here.

[0128] An embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the data synchronization method provided in any of the above embodiments is implemented.

[0129] The storage medium containing computer executable instructions provided in the embodiments of the present application can be used to store computer executable instructions of the data synchronization method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0130] An embodiment of the present application provides a computer program product, including computer instructions, which are used to enable a computer to execute the data synchronization method provided in the aforementioned embodiment.

[0131] The embodiments of the present application provide a computer program product that can be used to execute computer instructions of the data synchronization method provided in the aforementioned embodiments. The implementation method and principle are the same and will not be repeated here.

[0132] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0135] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0136] A part of the present application may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of computer program instructions in computer-readable media includes, but is not limited to, source files, executable files, installation package files, etc., and accordingly, the way in which computer program instructions are executed by a computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to the computer.

[0137] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example for illustration. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. The specific working process of the device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data synchronization method, characterized in that: include: Creating a shared memory at the primary site to intercept initial synchronization data sent by a plurality of cloud hosts at the primary site based on the shared memory; Cache the initial synchronization data to a preset cache storage pool, and record index information of the initial synchronization data; the index information includes a virtual disk offset, a data length, and a cache disk offset; When the cache condition of the preset cache storage pool meets the preset synchronization condition, generating a bitmap file of the initial synchronization data according to the index information of the initial synchronization data; Generate target synchronization data according to the bitmap file, and synchronize the target synchronization data to the backup site; wherein the data volume of the target synchronization data is smaller than the data volume of the initial synchronization data; The method further comprises: According to a preset identification cycle, a data consistency identification is issued to the shared memory, and the data consistency identification is added to the tail of the initial synchronization data intercepted at the corresponding time; In the process of caching the initial synchronization data to a preset cache storage pool, when the data consistency identifier carried at the end of the initial synchronization data is identified, the cache disk is split in the preset cache storage pool to replace the cache disk to store subsequent initial synchronization data, so that the initial synchronization data cached by the same cache disk belongs to the same preset identifier period.

2. The method according to claim 1, characterized in that: The method further comprises: When the cache disk is replaced, it is determined that the cache status of the preset cache storage pool meets the preset synchronization condition, so as to synchronize the cached data of the previous cache disk to the backup site.

3. The method according to claim 1, characterized in that: The step of caching the initial synchronization data into a preset cache storage pool includes: Obtaining a write fixed unit length of the preset cache storage pool; The initial synchronization data is cached in a preset cache storage pool according to the write fixed unit length.

4. The method according to claim 3, characterized in that The step of caching the initial synchronization data into a preset cache storage pool according to the write fixed unit length includes: According to the write fixed unit length, the initial synchronization data is divided and spliced ​​to record the initial synchronization data as a plurality of data to be written; wherein the length of each piece of the data to be written is equal to the write fixed unit length; The plurality of to-be-written data are cached in a preset cache storage pool.

5. The method according to claim 1, characterized in that The step of generating a bitmap file of the initial synchronization data according to the index information of the initial synchronization data comprises: Determine the virtual disk offset and data length of each piece of the initial synchronization data according to the index information of the initial synchronization data; According to the sending order of each piece of the initial synchronization data, and based on the virtual disk offset and data length of each piece of the initial synchronization data, a bitmap file of the initial synchronization data is generated.

6. The method according to claim 1, characterized in that The step of generating target synchronization data according to the bitmap file comprises: According to the bitmap file, all the initial synchronization data in the previous cache disk are deduplicated and merged to obtain target synchronization data.

7. The method according to claim 1, characterized in that The index information recording the initial synchronization data includes: Obtaining a virtual disk offset, data length, and cache disk offset of each piece of the initial synchronization data; Recording index information of the initial synchronization data according to the virtual disk offset, data length and cache disk offset of each piece of the initial synchronization data; The initial synchronization data is sent from the virtual disk to the shared memory by the cloud host, and the initial synchronization data is cached in the cache disk of the preset cache storage pool.

8. The method according to claim 1, characterized in that: The method further comprises: Before the cloud host on the primary site sends the initial synchronization data, a freeze instruction is sent to the cloud host, so that the cloud host will not modify the data during the process of sending the initial synchronization data.

9. The method according to claim 1, characterized in that: The method further comprises: According to the preset application consistency logic, an application consistency identifier is issued to the shared memory, and the application consistency identifier is added to the tail of the initial synchronization data intercepted at the corresponding time; In the process of caching the initial synchronization data to a preset cache storage pool, when the application consistency identifier carried at the end of the initial synchronization data is identified, the cache disk is split in the preset cache storage pool to replace the cache disk to store subsequent initial synchronization data, so that the initial synchronization data cached by the same cache disk belongs to the same application.

10. The method according to claim 9, characterized in that The method further comprises: The recovery point of the target synchronization data is determined according to the sending time of the application consistency identifier represented by the cache disk segmentation result of the preset cache storage pool, so that the backup site performs service recovery on the target synchronization data according to the recovery point.

11. A data synchronization system, characterized in that: include: A primary site and a backup site, wherein the primary site is deployed with several cloud hosts; The primary site synchronizes the initial synchronization data sent by the cloud host to the backup site based on the data synchronization method according to any one of claims 1 to 10.

12. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the method according to any one of claims 1 to 10.

Images