Dynamic adjustment method and device for synchronous data and electronic equipment

Abstract

The embodiment of the invention provides a dynamic adjustment method and device for synchronous data and electronic equipment, and the method comprises the steps: checking a bitmap file used for data synchronization, and judging whether a plurality of adjacent bits needing to be synchronized exist in the bitmap file or not; if a plurality of adjacent bits needing to be synchronized exist in the bitmap file, IO bytes of single-time synchronization data are dynamically adjusted according to the number proportion of the bits needing to be synchronized; and according to the IO bytes of the dynamically adjusted single synchronization data, carrying out data synchronization on the data needing to be synchronized. The method solves the problems that in the prior art, data synchronization cannot be rapidly completed, and performance and reliability cannot be considered at the same time.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a method for dynamically adjusting synchronized data, a device for dynamically adjusting synchronized data, an electronic device, and a computer-readable medium. Background Art

[0002] The data synchronization process is used to synchronize data between nodes, such as Figure 1 As shown, the data of base dev on the slave node is made consistent with the data of base dev on the master node. The specific synchronization process is as follows:

[0003] Check bitmap: Check the bitmap table / file in the master node memory and mark the bit value 1 as RESYNC (indicating that synchronization is required);

[0004] Reading data: After the system detects that the base dev status of the slave node is normal (available), it submits a read request to the master base dev according to the bit that needs to be synchronized in the bitmap table and the fixed IO size of a single bit (such as 1M) to read out the data that needs to be synchronized;

[0005] Write data: If the master base dev data is read successfully, submit a write request to other base devs according to the fixed IO size of a single bit (such as 1M) to write the read data to other base devs;

[0006] Waiting for return: After the synchronization is completed, the RESYNC mark is cleared. If there is a write request return failure, the bit is marked as 1 again and wait for the next data synchronization.

[0007] After data synchronization and data verification are completed, the corresponding DEV status will change from Degraded to online;

[0008] During the data synchronization process, if the master node fails or the master node storage pool fails, business will be interrupted.

[0009] Problems with existing solutions: During data synchronization, if the master node fails / the master node storage pool fails, it will cause business interruption; data synchronization has different degrees of impact on the reading and writing of front-end business, involving reading data from the master node, writing to the backup node, and accessing the bitmapfile file; how to quickly complete data synchronization while taking into account performance and reliability has become an urgent problem that needs to be solved. Summary of the invention

[0010] The embodiments of the present invention provide a method, device, electronic device and computer-readable storage medium for dynamically adjusting synchronized data to solve the problem in the prior art that data synchronization cannot be completed quickly and performance and reliability cannot be taken into account.

[0011] An embodiment of the present invention discloses a method for dynamically adjusting synchronization data, including:

[0012] Check the bitmap file used for data synchronization to determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file;

[0013] If there are multiple adjacent bits that need to be synchronized in the bitmap file, the IO bytes of a single synchronization data are dynamically adjusted according to the ratio of the number of bits that need to be synchronized;

[0014] According to the dynamic adjustment of the IO bytes of single synchronization data, the data to be synchronized is synchronized.

[0015] Optionally, a bitmap file used for data synchronization is checked to determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file, including:

[0016] Determine whether the values ​​of adjacent bits in the bitmap file are all 1. If so, there are multiple adjacent bits that need to be synchronized;

[0017] The bitmap file stores data marking bits that the master data device needs to synchronize with the slave data device during data synchronization.

[0018] Optionally, if there are multiple adjacent bits that need to be synchronized in the bitmap file, the IO bytes of a single synchronization data are dynamically adjusted according to the ratio of the number of bits that need to be synchronized, including:

[0019] If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than a%, then the number of IO bytes of single synchronization data is adjusted according to the multiple adjacent bits that need to be synchronized and the number of IO bytes corresponding to each bit, so that the adjusted number of IO bytes of single synchronization data is substantially the same as the number of all data bytes of the bits that need to be synchronized;

[0020] If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than b%, adjust the IO bytes of single synchronization data to the maximum IO bytes;

[0021] b is greater than a and greater than 10.

[0022] Optionally, data to be synchronized is synchronized according to the dynamically adjusted IO bytes of single synchronization data, including:

[0023] When the current master data device load is less than a first specified threshold, data to be synchronized is synchronized;

[0024] Alternatively, according to the dynamic adjustment of the IO bytes of the synchronization data, the data to be synchronized in the master data device is cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold;

[0025] Alternatively, the bitmap file and the data to be synchronized in the master data device are cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold.

[0026] Optionally, synchronizing the data to be synchronized includes:

[0027] When the slave data device is normal, the master data device actively pushes synchronization data to the slave data device when the load is less than the first specified threshold.

[0028] or,

[0029] When the slave data device is normal, the slave data device sends a data synchronization request to the master data device, and the master data device sends the synchronization data to the slave data device according to the data synchronization request;

[0030] or,

[0031] When the slave data device is normal and the master data device fails, the data to be synchronized buffered in the buffer of the master data device is synchronized to the slave data device according to the data replication mechanism.

[0032] The embodiment of the present invention also discloses a device for dynamically adjusting synchronization data, including:

[0033] A checking module is used to check the bitmap file used for data synchronization and determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file;

[0034] The dynamic adjustment module is used to dynamically adjust the IO bytes of a single synchronization data according to the ratio of the number of bits that need to be synchronized when there are multiple adjacent bits that need to be synchronized in the bitmap file;

[0035] The synchronization module is used to synchronize the data that needs to be synchronized according to the dynamic adjustment of the IO bytes of the single synchronization data.

[0036] Optionally, a dynamic adjustment module is specifically used to adjust the number of IO bytes of single synchronization data according to the number of IO bytes corresponding to the multiple adjacent bits that need to be synchronized, if the proportion of the multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than a%, so that the number of IO bytes of the single synchronization data after adjustment is substantially the same as the number of bytes of all data bits that need to be synchronized;

[0037] If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than b%, adjust the IO bytes of single synchronization data to the maximum IO bytes;

[0038] b is greater than a and greater than 10.

[0039] Optionally, a synchronization module is specifically used to

[0040] When the current master data device load is less than a first specified threshold, data to be synchronized is synchronized;

[0041] Alternatively, according to the dynamic adjustment of the IO bytes of the synchronization data, the data to be synchronized in the master data device is cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold;

[0042] Alternatively, the bitmap file and the data to be synchronized in the master data device are cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold.

[0043] The embodiment of the present invention further discloses an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0044] The memory is used to store computer programs;

[0045] The processor is used to implement the method described in the embodiment of the present invention when executing the program stored in the memory.

[0046] The embodiment of the present invention further discloses one or more computer-readable media on which instructions are stored. When executed by one or more processors, the processors are enabled to execute the method described in the embodiment of the present invention.

[0047] The embodiments of the present invention include the following advantages:

[0048] The method for dynamically adjusting synchronization data in an embodiment of the present invention checks adjacent bits during data synchronization to determine the proportion of bits that need to be synchronized, and then dynamically adjusts the IO bytes of word synchronization data. This allows more bytes to be synchronized at one time, avoiding the problem of multiple frequent readouts and accesses of a single bit in the prior art, slow synchronization, and causing the entire system to be in a sub-healthy Degraded state for event processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 It is an architectural diagram of the data synchronization process in the prior art;

[0050] Figure 2 is a flowchart of a method for dynamically adjusting synchronization data provided in an embodiment of the present invention;

[0051] Figure 3 is a flowchart of a method for dynamically adjusting synchronization data provided in an embodiment of the present invention;

[0052] Figure 4 It is a framework diagram of master-slave device data synchronization provided in an embodiment of the present invention;

[0053] Figure 5 is a flowchart of master-slave device data synchronization provided in an embodiment of the present invention;

[0054] Figure 6 is a partial step flow chart of a method for dynamically adjusting synchronization data provided in an embodiment of the present invention;

[0055] Figure 7 is a structural block diagram of a device for dynamically adjusting synchronous data provided in an embodiment of the present invention;

[0056] Figure 8 is a block diagram of an electronic device provided in an embodiment of the present invention;

[0057] Fig. 9 is a schematic diagram of a computer-readable medium provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0059] There are many ways to synchronize data within a system and between systems. Depending on the application scenario and requirements, the commonly used data synchronization mechanisms are:

[0060] Active push synchronization: A common mechanism is to pass data from one node to other nodes by push. When the data of the source node changes, it will actively push the update to other nodes to ensure data synchronization.

[0061] Active pull synchronization: Another mechanism is to synchronize data through pull. Nodes actively request data updates from other nodes periodically or as needed to ensure data consistency.

[0062] Subscription mode: This is a messaging mode in which nodes can publish messages and subscribe to messages of interest. When a node publishes new data, the subscriber node will receive the new data information of the node and process it accordingly.

[0063] Database replication: For distributed database systems, data replication can be used to synchronize data between different nodes. This mechanism can ensure data integrity and consistency, and provide fault tolerance and availability between different nodes. However, the data replication process will occupy system resources (CPU, bandwidth, etc.), which will have varying degrees of impact on front-end business.

[0064] There are also other technical solutions in the industry, such as using logs, queues, or distributed transactions to synchronize data between nodes. Choosing an appropriate mechanism requires considering factors such as system performance, consistency requirements, data volume, and network latency.

[0065] The method for dynamically adjusting synchronized data according to the embodiment of the present invention is applicable to the above-mentioned various data synchronization scenarios.

[0066] Reference Figure 2 , shows a flowchart of a method for dynamically adjusting synchronization data provided in an embodiment of the present invention, which may specifically include the following steps:

[0067] 201. Check the bitmap file used for data synchronization to determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file;

[0068] 202. If there are multiple adjacent bits that need to be synchronized in the bitmap file, the IO bytes of the single synchronization data are dynamically adjusted according to the ratio of the number of bits that need to be synchronized;

[0069] 203. According to the dynamic adjustment of the IO bytes of the single synchronization data, the data to be synchronized is synchronized.

[0070] The method of this embodiment checks adjacent bits during the data synchronization process to determine the proportion of bits that need to be synchronized, and then dynamically adjusts the IO bytes of the word synchronization data, thereby achieving synchronization of more bytes at a time, avoiding the problem of multiple frequent readouts and accesses of a single bit in the prior art, slow synchronization, and causing the entire system to be in a sub-healthy Degraded state.

[0071] For example, the above step 201 may further include the following sub-step 2011 which is not shown in the figure:

[0072] 2011. Determine whether the values ​​of adjacent bits in the bitmap file are all 1. If so, there are multiple adjacent bits that need to be synchronized;

[0073] The bitmap file stores data marking bits that the master data device needs to synchronize with the slave data device during data synchronization.

[0074] Generally, in data synchronization, bitmap files and bit values ​​are widely used to efficiently manage and track data changes. A bitmap file is a data structure used to represent a set of Boolean values ​​(0 or 1), where each bit usually corresponds to the state of a data item. It is very useful in areas such as incremental synchronization, data replication, and backup.

[0075] For example, for incremental synchronization, only the data that has changed since the last synchronization is synchronized, rather than all the data, to improve efficiency. A Bitmap file is used to record the change status of each data item. Each bit corresponds to a data item, 0 means no change, and 1 means changed.

[0076] For data replication, the purpose is to maintain data consistency between multiple nodes. Use a bitmap file to record which data items have been successfully replicated to the target node and which data items still need to be synchronized. During the backup process, only the data that has changed since the last backup is backed up. Use a bitmap file to record which files or data blocks have been backed up and which need to be backed up.

[0077] The meaning of the bit: 0 means the corresponding data item has not changed or is not synchronized, 1 means the corresponding data item has changed or needs to be synchronized. The index of each bit corresponds to a unique identifier of a data item (such as a file ID, record ID, etc.).

[0078] In actual use, initialization can create a bitmap file, and all bits are initialized to 0, indicating that all data items have not changed. When the data items change, the corresponding bit positions are set to 1. During the synchronization process, the bitmap file is traversed to find all bits with a value of 1, and the corresponding changed data items are synchronized to the target node. After the synchronization is completed, the bits corresponding to the synchronized data items are reset to 0. In this embodiment, after finding all bits with a value of 1, it can be determined whether the values ​​of adjacent bits in the bitmap file are all 1. If so, there are multiple adjacent bits that need to be synchronized. The proportion of the number of bits that need to be synchronized is dynamically adjusted, and the IO bytes of a single synchronization data are adjusted. This can avoid frequent reading and access during the synchronization process, affecting the foreground business, and also allow the system to handle the sub-healthy Degraded state for a long time.

[0079] Reference Figure 3 , shows a flowchart of a method for dynamically adjusting synchronization data provided in an embodiment of the present invention, which may specifically include the following steps:

[0080] 301. Determine whether the values ​​of adjacent bits in the bitmap file are all 1. If so, there are multiple adjacent bits that need to be synchronized.

[0081] 302. If there are multiple adjacent bits that need to be synchronized in the bitmap file, determine the proportion of the multiple adjacent bits that need to be synchronized relative to the entire bitmap file;

[0082] 303. If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than a%, then according to the multiple adjacent bits that need to be synchronized and the number of IO bytes corresponding to each bit, the number of IO bytes of single synchronization data is adjusted so that the adjusted number of IO bytes of single synchronization data is substantially the same as the number of bytes of all data bits that need to be synchronized;

[0083] If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than b%, adjust the IO bytes of single synchronization data to the maximum IO bytes;

[0084] b is greater than a and greater than 10. For example, b can be 80 and a can be 20.

[0085] 304. When the current load of the master data device is less than a first specified threshold, synchronize the data that needs to be synchronized.

[0086] For example, the data to be synchronized in the master data device may be cached in the buffer layer according to the dynamically adjusted IO bytes of the synchronization data, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer layer may be synchronized; the second specified threshold is less than the first specified threshold;

[0087] In another implementation, the bitmap file and the data that need to be synchronized in the main data device can also be cached in the buffer layer. When the current main data device load is less than a second specified threshold, the data that needs to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold.

[0088] In a possible implementation, when the slave data device is normal, the master data device actively pushes synchronization data to the slave data device when the load is less than a first specified threshold.

[0089] In a second possible implementation, when the slave data device is normal, the slave data device sends a data synchronization request to the master data device, and the master data device sends synchronization data to the slave data device according to the data synchronization request;

[0090] In a third possible implementation, when the slave data device is normal and the master data device fails, the to-be-synchronized data buffered in the buffer of the master data device is synchronized to the slave data device according to a data replication mechanism.

[0091] The method of this embodiment improves reliability while improving performance, balancing reliability and performance. The above method solves the problem in the prior art that if a master node failure / master node storage pool failure occurs during data synchronization, business interruption will occur; at the same time, it solves the problem that the reading and writing of the foreground business will be affected to varying degrees during the data synchronization process, and achieves the purpose of quickly completing data synchronization while taking into account performance and reliability.

[0092] In this embodiment, high-performance storage nodes and high-performance access mechanisms are adopted. For example, the values ​​of adjacent bits are compared to automatically adjust the IO size of the synchronized data, that is, the IO size of reading data from the master bdev and writing data to the slave bdev; if the values ​​of multiple bits appear continuously and are all 1, then the IO size = the size of the space corresponding to the multiple bits, and the data is read and written at one time, which improves the synchronization speed and shortens the synchronization time.

[0093] Combined with existing technologies, the effective time of this mechanism can be set according to the time period (select the time period with low business pressure for the front-end business to start), reduce the proportion of system performance degradation, reduce the duration of the impact on system performance, and quickly complete data synchronization. All nodes are in normal state and can process business normally, improving the overall availability of the system. Through this mechanism, frequent reading and access are avoided, which affects the performance of the front-end business. If the synchronization is slow, the system will be in a sub-healthy Degraded state for a long time, and it will be unusable after a second failure. Specifically, if Figure 4 and Figure 5 As shown, Figure 4 A framework diagram of master-slave device data synchronization provided in an embodiment of the present invention is shown; Figure 5 The method shown may include the following steps:

[0094] 501. When both the master data device (dev) and the slave data device are online, the business can be written normally. At this time, there is no need to set the bitmap flag;

[0095] 502. When the slave data device (dev) is not online, during normal business write operations, only the data that is successfully written to the master data device needs to set a flag bit in the corresponding bitmap (the flag bit is the bit);

[0096] 503. After the slave data device recovers to normal, data synchronization is started to synchronize the data of the master data device corresponding to the bitmap flag bit 1 to the slave data device.

[0097] For example, check the bitmap file used for data synchronization to determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file. If there are multiple adjacent bits that need to be synchronized in the bitmap file, dynamically adjust the IO bytes of single synchronization data according to the proportion of the number of bits that need to be synchronized; IO bytes refer to data transmitted in bytes during input and output (I / O) operations, and synchronize the data that needs to be synchronized.

[0098] The method of this embodiment targets different bitmap scenarios (such as data synchronization between controllers, data synchronization between nodes, differentiated data synchronization, etc.), and dynamically calculates the size of single synchronization data according to the actual amount and characteristics of data to be synchronized. Compared with a fixed data synchronization size, it can reduce the number of data reading and writing, improve the synchronization speed, shorten the time required for synchronization, and reduce the impact on foreground business reading and writing.

[0099] In actual applications, the bitmap flag of the master node is detected, and all flag bits are first traversed and counted: if the bit with the flag bit set to 1 is less than 20%, synchronization is performed according to the fixed synchronization size; if the bit with the flag bit set to 1 is greater than 20%, synchronization is performed according to the dynamically calculated synchronization size;

[0100] If the bit with the flag bit set to 1 is greater than 100%, read from the primary dev according to the maximum IO size (such as 16M) and write 16M to the backup dev.

[0101] That is, by determining the proportion of a plurality of adjacent bits that need to be synchronized relative to the entire bitmap file, a synchronization mode is selected, thereby enhancing the flexibility of the method for dynamically adjusting synchronization data of this embodiment and improving synchronization efficiency.

[0102] See also Figure 6 , shows a flowchart of dynamically calculating the synchronization size of data between replicas in a method for dynamically adjusting synchronization data provided in an embodiment of the present invention, which may specifically include the following steps:

[0103] 601. Traverse and count the proportions of multiple adjacent bits that need to be synchronized relative to the entire bitmap file, that is, count the proportions of multiple adjacent bits whose flag bits are 1 relative to the entire bitmap file;

[0104] 602. If the proportion of multiple adjacent bits to be synchronized to the entire bitmap file is less than 20% (here 20% is taken as an example and limited), synchronization is performed according to a preset fixed synchronization size.

[0105] 603. If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than or equal to 20% and less than 100%, the number of IO bytes of single synchronization data is adjusted according to the multiple adjacent bits that need to be synchronized and the number of IO bytes corresponding to each bit, so that the adjusted number of IO bytes of single synchronization data is basically the same as the number of all data bytes of the bits that need to be synchronized; that is, synchronization is performed according to the dynamically calculated synchronization size.

[0106] 604. If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is equal to 100%, the number of IO bytes of single synchronization data is adjusted to a preset maximum number of IO bytes.

[0107] This embodiment adopts a high-performance storage node and a high-performance access mechanism, and automatically adjusts the IO size of the synchronization data by comparing the values ​​of adjacent bits and the ratio of multiple adjacent bits that need to be synchronized to the entire bitmap file, that is, the IO size of reading data from the master data device and writing data to the slave data device; if the values ​​of multiple bits appear continuously and are all 1, then the IO size is the size of the space corresponding to the multiple bits, and the reading and writing are done at one time, which improves the synchronization speed and shortens the synchronization time; and by dynamically adjusting the IO size of the synchronization data, the flexibility of synchronization is enhanced, and the synchronization efficiency is further improved.

[0108] For example, synchronization is performed according to a dynamically calculated synchronization size, which may specifically include the following steps:

[0109] Start traversing the plan from the first bit, find the position n where the first bit value appears to be 1, and traverse the positions one by one (n+1, n+2, n+3, etc.) to find the position where 0 appears. If the IO size corresponding to 1 bit is 64K, then the amount of data to be synchronized corresponding to the consecutive bits that are 1 is i*64K. This size is used as the IO size for reading data from the master data device and writing data to the slave data device.

[0110] For example, if a bitmap file is as shown in Table 1, where the first occurrence of the third bit is 1, the values ​​of the fourth, fifth, and sixth bits are all 1, and the value of the seventh bit is 0, then n = 3, i = 4, then the IO read and write size of the corresponding field for the first data synchronization should be 4*64K;

[0111] Traversing further, the value of the 8th bit is 1, but the 9th bit is 0, so n=8, i=1, and the IO read and write size of the corresponding field of the second data synchronization should be 1*64K;

[0112] Continue to traverse, the value of the 10th bit is 1, the value of the 11th bit is also 1, and the value of the 12th bit is 0, so n = 10, i = 2, then the IO read and write size of the corresponding field of the third data synchronization should be 2*64K;

[0113] Similarly, obtain the subsequent IO read and write sizes.

[0114] According to the calculated IO size, data is read from the master data device and written to the slave data device. The above process is repeated until data synchronization is completed and all bit values ​​are 0, that is, the data on the master data device and the slave data device are consistent.

[0115] Table 1: bitmap file

[0116] bit 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 … 0 0 1 1 1 1 0 1 0 1 1 0 0 1 1 0 0 …

[0117] This embodiment dynamically calculates the size of a single synchronization data for different bitmap scenarios according to the actual amount and characteristics of data to be synchronized. Compared with a fixed data synchronization size, it can reduce the number of data reading and writing, improve the synchronization speed, shorten the time required for synchronization, and reduce the impact on foreground business reading and writing.

[0118] Since the data synchronization process will have a certain impact on the front-end business, the specific impacts are as follows: the data synchronization process may occupy system resources, such as network bandwidth, computing resources, etc., resulting in performance degradation and affecting the response speed of the front-end business; if errors or abnormalities occur in data synchronization, data inconsistency or loss may occur, thereby affecting the normal operation of the front-end business. In order to reduce the impact of data synchronization on the front-end business, the following measures can be considered: reasonably plan the time of data synchronization to avoid data synchronization operations during business peaks; control the frequency and amount of data synchronization to avoid large-scale batch synchronization of data; use asynchronous processing mechanisms to isolate the data synchronization process from the front-end business to avoid direct impact on the front-end business.

[0119] For example, since data synchronization will affect the foreground service of the master data device, data synchronization can be performed when the load of the master data device is small, that is, when the current load of the master data device is less than the first specified threshold, the data that needs to be synchronized will be synchronized.

[0120] In an optional implementation, a buffer layer is set up to store the data to be synchronized. According to the dynamic adjustment of the IO bytes of the synchronization data, the data to be synchronized in the master data device is cached in the buffer layer. When the load of the master data device is small, the data to be synchronized in the buffer layer is synchronized. That is, when the current load of the master data device is less than a second specified threshold, the data to be synchronized in the buffer layer is synchronized; the second specified threshold is less than the first specified threshold.

[0121] In the second optional implementation, the bitmap file and the data to be synchronized in the primary data device are cached in the buffer layer, and when the current primary data device load is less than the second specified threshold, the data to be synchronized in the buffer layer is synchronized. That is, a buffer layer is added to back up a bitmap file, which is used for data synchronization when the primary data device load is low.

[0122] In a third optional implementation, the availability of this embodiment is provided by a backup measure, that is, a buffer layer is added to store the data to be synchronized, so that in the event of a data failure of the primary data device, data synchronization and foreground business reading and writing will not be affected.

[0123] In the fourth optional implementation method, a buffer layer is added to back up a bitmap file. In the event that the bitmap file of the primary data device is abnormally inaccessible, data synchronization and foreground business reading and writing will not be affected.

[0124] In a fifth optional implementation, during the data synchronization process, data can be directly read from the buffer layer to avoid affecting the foreground business read and write operations of the master data device.

[0125] This embodiment provides a method for dynamically adjusting synchronization data. For different bitmap scenarios, the size of single synchronization data is dynamically calculated according to the actual amount and characteristics of data to be synchronized. Compared with a fixed data synchronization size, the number of data reads and writes can be reduced, the synchronization speed can be improved, the time required for synchronization can be shortened, and the impact on the reading and writing of foreground services can be reduced. On this basis, a buffer layer for storing bitmap and data to be synchronized is added, so that data synchronization can continue when the original bitmap file is inaccessible, the master data device fails, or the foreground service of the master data device is busy.

[0126] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0127] Reference Figure 7 , shows a structural block diagram of a device for dynamically adjusting synchronization data provided in an embodiment of the present invention, which may specifically include the following modules:

[0128] A checking module is used to check the bitmap file used for data synchronization and determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file;

[0129] The dynamic adjustment module is used to dynamically adjust the IO bytes of a single synchronization data according to the ratio of the number of bits that need to be synchronized when there are multiple adjacent bits that need to be synchronized in the bitmap file;

[0130] The synchronization module is used to synchronize the data that needs to be synchronized according to the dynamic adjustment of the IO bytes of the single synchronization data.

[0131] Furthermore, the checking module is specifically used to determine whether the values ​​of adjacent bits in the bitmap file are all 1. If so, there are multiple adjacent bits that need to be synchronized.

[0132] The bitmap file stores data marking bits that the master data device needs to synchronize with the slave data device during data synchronization.

[0133] In a possible implementation, the dynamic adjustment module of this embodiment is specifically used to:

[0134] If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than a%, then the number of IO bytes of single synchronization data is adjusted according to the multiple adjacent bits that need to be synchronized and the number of IO bytes corresponding to each bit, so that the adjusted number of IO bytes of single synchronization data is substantially the same as the number of all data bytes of the bits that need to be synchronized;

[0135] If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than b%, adjust the IO bytes of single synchronization data to the maximum IO bytes;

[0136] b is greater than a and greater than 10.

[0137] In a second possible implementation manner, the synchronization module of this embodiment is specifically used to:

[0138] When the current master data device load is less than a first specified threshold, data to be synchronized is synchronized;

[0139] Alternatively, according to the dynamic adjustment of the IO bytes of the synchronization data, the data to be synchronized in the master data device is cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold;

[0140] Alternatively, the bitmap file and the data to be synchronized in the master data device are cached in the buffer layer, and when the current load of the master data device is less than the second specified threshold, the data to be synchronized in the buffer is synchronized;

[0141] The second specified threshold is less than the first specified threshold.

[0142] In a third possible implementation, the synchronization module of this embodiment is further configured to, when the slave data device is normal and the master data device has a load less than a first specified threshold, actively push synchronization data to the slave data device, or, when the slave data device is normal, the slave data device sends a data synchronization request to the master data device, and the master data device sends the synchronization data to the slave data device according to the data synchronization request;

[0143] Alternatively, when the slave data device is normal and the master data device fails, the data to be synchronized buffered in the buffer of the master data device is synchronized to the slave data device according to the data replication mechanism.

[0144] The present embodiment provides a device for dynamically adjusting synchronization data. For different data synchronization scenarios, the size of single synchronization data is dynamically calculated according to the actual amount and characteristics of data to be synchronized. Compared with the fixed data synchronization size, the number of data reads and writes can be reduced, the synchronization speed can be improved, the time required for synchronization can be shortened, and the impact on the reading and writing of foreground services can be reduced. On this basis, a buffer layer for storing bitmap and data to be synchronized is added. In the case that the original bitmap file is inaccessible, the master data device fails, or the foreground service of the master data device is busy, data synchronization can continue.

[0145] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0146] In addition, an embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303 and a communication bus 1304, wherein the processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304.

[0147] Memory 1303, used for storing computer programs;

[0148] The processor 1301 is used to implement the following steps when executing the program stored in the memory 1303:

[0149] Check the bitmap file used for data synchronization to determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file; bitmap is a technology that efficiently stores and queries data through bit mapping. It uses a bit to mark the state of an element, which can effectively save memory space when processing large-scale data sets.

[0150] If there are multiple adjacent bits that need to be synchronized in the bitmap file, the IO bytes of the single synchronization data are dynamically adjusted according to the proportion of the number of bits that need to be synchronized; IO bytes refer to data transmitted in bytes during the input and output (I / O) operation process.

[0151] According to the dynamic adjustment of the IO bytes of single synchronization data, the data to be synchronized is synchronized.

[0152] Alternatively, the following steps are implemented: if the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than a%, then according to the multiple adjacent bits that need to be synchronized and the number of IO bytes corresponding to each bit, the number of IO bytes of single synchronization data is adjusted, so that the adjusted number of IO bytes of single synchronization data is substantially the same as the number of all data bytes of the bits that need to be synchronized;

[0153] If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than b%, adjust the IO bytes of single synchronization data to the maximum IO bytes;

[0154] b is greater than a and greater than 10.

[0155] Alternatively, when the current master data device load is less than the first specified threshold, data to be synchronized is synchronized;

[0156] Alternatively, according to the dynamic adjustment of the IO bytes of the synchronization data, the data to be synchronized in the master data device is cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold;

[0157] Alternatively, the bitmap file and the data to be synchronized in the main data device are cached in the buffer layer, and when the current main data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold.

[0158] In this embodiment, for data synchronization (including metadata and basic data), the size of single synchronization data is dynamically calculated according to the actual amount and characteristics of data to be synchronized. Compared with the fixed data synchronization size, the number of data reading and writing can be reduced, the synchronization speed can be improved, the time required for synchronization can be shortened, and the impact on the reading and writing of foreground services can be reduced. At the same time, it also avoids being in the synchronization state for a long time. If a failure occurs again, the entire system will be unavailable;

[0159] If a buffer is added to store the bitmap and data to be synchronized, data synchronization can continue when the original bitmap file is inaccessible or the main dev fails. The system will not be inaccessible and can continue to be used normally.

[0160] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0161] The communication interface is used for communication between the above terminal and other devices.

[0162] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0163] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0164] like Fig. 9 As shown, in another embodiment provided by the present invention, a computer-readable storage medium 1401 is also provided, in which instructions are stored. When the computer-readable storage medium 1401 is run on a computer, the computer executes the dynamic adjustment method of synchronization data described in the above embodiment.

[0165] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes the method for dynamically adjusting synchronization data described in the above embodiment.

[0166] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0167] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0168] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0169] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A method for dynamically adjusting synchronization data, characterized in that: include: Check the bitmap file used for data synchronization to determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file; If there are multiple adjacent bits that need to be synchronized in the bitmap file, the IO bytes of a single synchronization data are dynamically adjusted according to the ratio of the number of bits that need to be synchronized; According to the dynamic adjustment of the IO bytes of single synchronization data, the data to be synchronized is synchronized.

2. The method according to claim 1, characterized in that Check the bitmap file used for data synchronization to determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file, including: Determine whether the values ​​of adjacent bits in the bitmap file are all 1. If so, there are multiple adjacent bits that need to be synchronized; The bitmap file stores data marking bits that the master data device needs to synchronize with the slave data device during data synchronization.

3. The method according to claim 1, characterized in that If there are multiple adjacent bits in the bitmap file that need to be synchronized, the IO bytes of a single synchronization data are dynamically adjusted according to the ratio of the number of bits that need to be synchronized, including: If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than a%, then the number of IO bytes of single synchronization data is adjusted according to the multiple adjacent bits that need to be synchronized and the number of IO bytes corresponding to each bit, so that the adjusted number of IO bytes of single synchronization data is substantially the same as the number of all data bytes of the bits that need to be synchronized; If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than b%, adjust the IO bytes of single synchronization data to the maximum IO bytes; b is greater than a and greater than 10.

4. The method according to claim 1, characterized in that: According to the dynamic adjustment of the IO bytes of single synchronization data, the data to be synchronized is synchronized, including: When the current master data device load is less than a first specified threshold, data to be synchronized is synchronized; Alternatively, according to the dynamic adjustment of the IO bytes of the synchronization data, the data to be synchronized in the master data device is cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold; Alternatively, the bitmap file and the data to be synchronized in the master data device are cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold.

5. The method according to claim 1, characterized in that The data that needs to be synchronized includes: When the slave data device is normal, the master data device actively pushes synchronization data to the slave data device when the load is less than the first specified threshold. or, When the slave data device is normal, the slave data device sends a data synchronization request to the master data device, and the master data device sends the synchronization data to the slave data device according to the data synchronization request; or, When the slave data device is normal and the master data device fails, the data to be synchronized buffered in the buffer of the master data device is synchronized to the slave data device according to the data replication mechanism.

6. A device for dynamically adjusting synchronization data, characterized in that: include: A checking module is used to check the bitmap file used for data synchronization and determine whether there are multiple adjacent bits that need to be synchronized in the bitmap file; The dynamic adjustment module is used to dynamically adjust the IO bytes of a single synchronization data according to the ratio of the number of bits that need to be synchronized when there are multiple adjacent bits that need to be synchronized in the bitmap file; The synchronization module is used to synchronize the data that needs to be synchronized according to the dynamic adjustment of the IO bytes of the single synchronization data.

7. The dynamic adjustment device according to claim 6, characterized in that: The dynamic adjustment module is specifically used to adjust the number of IO bytes of single synchronization data according to the number of IO bytes corresponding to each bit position if the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than a%, so that the number of IO bytes of single synchronization data after adjustment is substantially the same as the number of bytes of all data that need to be synchronized bits; If the proportion of multiple adjacent bits that need to be synchronized relative to the entire bitmap file is greater than b%, adjust the IO bytes of single synchronization data to the maximum IO bytes; b is greater than a and greater than 10.

8. The dynamic adjustment device according to claim 6, characterized in that: Synchronization module, specifically used for When the current master data device load is less than a first specified threshold, data to be synchronized is synchronized; Alternatively, according to the dynamic adjustment of the IO bytes of the synchronization data, the data to be synchronized in the master data device is cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold; Alternatively, the bitmap file and the data to be synchronized in the master data device are cached in the buffer layer, and when the current master data device load is less than a second specified threshold, the data to be synchronized in the buffer is synchronized; the second specified threshold is less than the first specified threshold.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; The memory is used to store computer programs; The processor is used to implement the method according to any one of claims 1 to 5 when executing the program stored in the memory.

10. A computer-readable medium having instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method according to any one of claims 1 to 5.

Images