Method and device for intelligently switching full-amount synchronization and incremental synchronization

Abstract

The invention relates to a method and a device for intelligently switching full-amount synchronization and incremental synchronization. The method comprises the following steps: during primary synchronization, completely transmitting all data of a source end to a target end by adopting a full-quantity synchronization mode; monitoring a data change condition in real time through a timestamp or a database log, and obtaining a change data volume; dynamically selecting an incremental synchronization mode or a full-amount synchronization mode according to the proportion of the changed data volume to the total data volume; in the incremental synchronization process, if synchronization fails or database file analysis errors occur, the mode is automatically switched to a full-amount synchronization mode; and full-amount synchronization operation is executed regularly so as to ensure the integrity and consistency of the data. Through the intelligent switching synchronization mode, the data synchronization efficiency is remarkably improved, and the data transmission quantity and the resource consumption are reduced; meanwhile, a fault-tolerant mechanism and regular full-amount synchronous operation are combined, and long-term consistency and reliability of data are guaranteed. The scheme is suitable for various business scenes, and has wide applicability and flexibility.

Description

Technical Field

[0001] This application belongs to the technical field of data synchronization, and particularly relates to a method, device, computer-readable storage medium, and electronic device for intelligent switching between full synchronization and incremental synchronization. Background Art

[0002] Data synchronization technology is an important means to achieve data consistency among multiple data sources, and is generally divided into two methods: full synchronization and incremental synchronization.

[0003] Full synchronization means that all data in the source table is completely transmitted to the target end in one operation. This method is suitable for scenarios with a small amount of data or initial synchronization, because its operation is simple and can ensure data integrity. However, full synchronization has obvious limitations: it consumes a large amount of network bandwidth and memory resources. Especially when the data change ratio is low, a large amount of unchanged data will also be repeatedly synchronized, resulting in low efficiency.

[0004] Incremental synchronization only transmits the data that has changed in the source table (including the added, deleted, and modified parts), which can significantly reduce the data transmission volume and resource consumption. It is suitable for scenarios with a large amount of data and frequent changes, but its implementation is relatively complex and usually relies on technologies such as timestamps and database logs to identify data changes. In addition, incremental synchronization needs to process deleted and modified data, and the data consistency problem is also one of its key challenges.

[0005] Currently, in actual application scenarios, simply adopting the full synchronization or incremental synchronization method often fails to meet the requirements. For example, the order and inventory management system of an e-commerce platform needs to synchronize a large amount of order and inventory data. The order data is updated frequently, while the inventory data is updated relatively less. If only full synchronization is used, it will lead to resource waste and low efficiency; if only incremental synchronization is used, it may not be able to handle scenarios where data changes infrequently. Therefore, traditional data synchronization methods are difficult to achieve a balance among efficiency, resource consumption, and data consistency.

[0006] In view of the limitations of existing data synchronization technologies, there is an urgent need for a new solution that can balance the efficiency, resource consumption, and data consistency of data synchronization in scenarios with different data characteristics (such as update frequency, data volume, etc.). For example, by combining the advantages of full synchronization and incremental synchronization, or introducing new synchronization mechanisms and technologies to adapt to complex and changeable actual application scenarios. Summary of the Invention

[0007] To address the above problems, this application proposes a new method for intelligent switching between full synchronization and incremental synchronization, aiming to solve the problem that traditional synchronization methods cannot balance the efficiency, resource consumption, and data consistency of data synchronization.

[0008] This solution aims to achieve efficient, consistent, and resource-optimized data synchronization by dynamically selecting the synchronization method. The core of this solution lies in intelligently switching between full synchronization and incremental synchronization modes based on factors such as the amount of data changes, synchronization frequency, and resource consumption. The main implementation process is as follows: 1. Initial synchronization During the initial synchronization, the full synchronization method is adopted to transfer all the data at the source end to the target end completely, ensuring the complete consistency of the data between the target end and the source end, and laying a foundation for subsequent synchronization operations.

[0009] 2. Data change monitoring Monitor the data changes in real time through timestamps (such as the update_time field) or log files (such as the Binlog log of MySQL). The monitoring mechanism can accurately identify data addition, deletion, or modification operations and record the relevant change information, providing a basis for the subsequent selection of the synchronization method.

[0010] 3. Dynamic synchronization method selection Based on the ratio of the monitored changed data volume to the total data volume, dynamically select the synchronization method according to the following principles: When the data change ratio is relatively low, preferentially select incremental synchronization to reduce the data transmission volume and resource consumption; When the data change ratio is relatively high or close to the full amount of data, switch to full synchronization to avoid efficiency problems caused by repeated synchronization.

[0011] 4. Regular full synchronization To ensure data consistency, perform a full synchronization operation regularly (such as weekly or monthly). This mechanism can correct data deviations or cumulative errors that may occur during the incremental synchronization process, ensuring the long-term consistency of the data between the target end and the source end.

[0012] 5. Fault tolerance mechanism During the incremental synchronization process, if a synchronization failure or database file parsing error occurs, the system will automatically switch to the full synchronization mode to ensure data integrity and consistency. The fault tolerance mechanism can effectively handle abnormal situations and ensure the reliability of data synchronization.

[0013] This solution realizes the dynamic balance among efficiency, resource consumption, and data consistency in scenarios with different data characteristics (such as data volume and update frequency) by intelligently switching the synchronization method. At the same time, through the monitoring mechanism, dynamic selection strategy, and fault tolerance mechanism, the flexibility and reliability of data synchronization are improved, making it applicable to a variety of complex application scenarios.

[0014] Specifically, this application provides the following technical solutions: The first aspect of the present application provides a method for intelligent switching between full synchronization and incremental synchronization. The method includes: S1. At the initial synchronization, adopt the full synchronization method to completely transfer all data at the source end to the target end to ensure that the data at the target end is exactly the same as that at the source end; S2. Monitor the change situation of data in real time through timestamps or database logs to obtain the amount of changed data; S3. During the incremental synchronization process, dynamically select the incremental synchronization or full synchronization method according to the ratio of the amount of changed data to the total amount of data; S4. When incremental synchronization fails or there is an error in parsing the database file, automatically switch to the full synchronization mode.

[0015] Further, in step S3 of the method of the present application, it also includes: Compare the ratio of the amount of changed data to the total amount of data (data change ratio) with a preset threshold: when the data change ratio is lower than the preset threshold, select incremental synchronization; when the data change ratio is higher than the preset threshold, switch to full synchronization.

[0016] Further, in the method of the present application, the preset threshold is set according to business requirements. A higher threshold is set for business scenarios with high real-time requirements; a lower threshold is set for business scenarios with high data consistency requirements.

[0017] Further, the method of the present application also includes: S5. Regularly perform full synchronization operations to ensure data integrity and consistency; the period for regularly performing full synchronization operations is set according to business requirements.

[0018] Further, in the method of the present application, the timestamp includes the creation time field and update time field of the source table, and the amount of changed data within a certain time range is obtained through the update time field.

[0019] Further, in the method of the present application, the database log is the Binlog log of MySQL, and the amount of data for the addition, deletion, and modification operations of the source table is obtained by parsing the Binlog log.

[0020] Further, in step S3 of the method of the present application, it also includes: making a comprehensive judgment based on the synchronization frequency and resource situation, and dynamically selecting the incremental synchronization or full synchronization method; (1) For high-frequency data synchronization scenarios, preferentially select incremental synchronization; For low-frequency data synchronization scenarios, select the synchronization method according to the data change ratio; (2) When the network bandwidth and memory resources are sufficient, preferentially select full synchronization; When network bandwidth and memory resources are insufficient, incremental synchronization is preferentially selected.

[0021] The second aspect of the present application provides a device for intelligent switching between full synchronization and incremental synchronization, and the device includes: An initial synchronization module, configured to, during initial synchronization, adopt a full synchronization method to completely transmit all data at the source end to the target end, ensuring that the data at the target end is exactly the same as that at the source end; A data change detection module, configured to monitor the data change situation in real time through timestamps or database logs to obtain the amount of changed data; An intelligent switching algorithm module, configured to, during the incremental synchronization process, dynamically select an incremental synchronization or full synchronization method according to the ratio of the amount of changed data to the total amount of data; A fault tolerance module, configured to automatically switch to the full synchronization mode when incremental synchronization fails or there is an error in parsing the database file; A periodic synchronization module, configured to perform full synchronization operations periodically to ensure data integrity and consistency.

[0022] When the device runs, it implements the steps of the foregoing method for intelligent switching between full synchronization and incremental synchronization.

[0023] The third aspect of the present application provides an electronic device, including: a memory and a processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the foregoing method for intelligent switching between full synchronization and incremental synchronization.

[0024] The fourth aspect of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the foregoing method for intelligent switching between full synchronization and incremental synchronization.

[0025] In summary, through the intelligent switching of the synchronization method, the present solution realizes the high efficiency of data synchronization, resource optimization, and data consistency guarantee. At the same time, it has a wide range of scenario adaptability. The present application has the following advantages: (1) Improve synchronization efficiency Through the intelligent switching mechanism, the present solution can preferentially adopt the incremental synchronization method when the amount of data change is small, significantly reducing the amount of data transmission, thereby shortening the synchronization time. This dynamic selection strategy optimizes the synchronization process to the greatest extent and improves the overall efficiency of data synchronization.

[0026] (2) Save resources In terms of resource consumption, the intelligent switching mechanism can dynamically select the synchronization method according to the data change situation, minimizing unnecessary data transmission to the greatest extent. Compared with the traditional full - volume synchronization method, this solution significantly reduces the occupation of network bandwidth and storage resources, effectively reducing the data transmission cost.

[0027] (3) Ensure data consistency This solution calibrates the data between the target end and the source end periodically through the regular full - volume synchronization mechanism to ensure data integrity and consistency. At the same time, the fault - tolerance mechanism can automatically switch to full - volume synchronization when the incremental synchronization fails or an exception occurs, further ensuring the reliability of data synchronization and effectively avoiding data deviation or loss.

[0028] (4) Adapt to diverse scenarios This solution has a high degree of flexibility and can dynamically select the synchronization method according to different business requirements. Whether it is data migration, real - time analysis, or backup and recovery scenarios, this application can meet diverse business needs through the intelligent switching mechanism and has wide applicability.

[0029] Other features and advantages of this application will be elaborated in detail in the subsequent description, or can be understood by implementing the relevant technical solutions of this application. The purpose and other advantages of this application can be achieved through the technical features and means clearly pointed out in the description, claims, and drawings, and obtained through the implementation process of these technical contents. Brief Description of the Drawings

[0030] In order to more clearly elaborate the technical solutions of the embodiments of this application, the drawings involved in the description of the embodiments will be briefly introduced below. It should be noted that the drawings only show some embodiments of this application. For those skilled in the art, other relevant drawings can be derived based on these drawings without creative efforts.

[0031] Figure 1 It is the overall implementation flowchart of the intelligent switching method for full - volume synchronization and incremental synchronization of this application.

[0032] Figure 2 It is the implementation flowchart of the intelligent switching algorithm in the method of this application.

[0033] Figure 3 It is the composition structure diagram of the device for intelligent switching between full - volume synchronization and incremental synchronization of this application.

[0034] Figure 4 It is the structural schematic diagram of the electronic device provided by the embodiment of this application. Detailed Description of the Embodiments

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer and more explicit, the following will, in conjunction with the accompanying drawings in the embodiments of this application, clearly and completely describe the technical solutions in the embodiments of this application. It should be clear that the described embodiments are only partial embodiments of this application, rather than all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the protection scope of this application.

[0036] In this article, the term "including" and any form of its deformation (such as "including", "including") are open expressions and should be understood as "including but not limited to", that is, the listed content is not an exhaustive list and may also include other content not explicitly mentioned. The term "based on" should be understood as "at least partially based on", that is, the referred basis or condition may not be the only factor and may also involve other relevant factors. The term "an embodiment" should be understood as "at least one embodiment", that is, the described embodiment is not the only possible implementation, and there may also be other similar embodiments.

[0037] In this application, when the terms "one" and "multiple" are used to modify relevant elements or features, their expressions are illustrative rather than restrictive. Unless otherwise clearly stated in the context, "one" should be understood as "at least one", and "multiple" should be understood as "at least two". Those skilled in the art should reasonably interpret these terms according to the semantic and logical relationships in the context to ensure that they cover the possibility of "one or more".

[0038] Figure 1 The following shows the overall implementation process of the full synchronization and incremental synchronization intelligent switching method provided by this application, including the following steps: S1. During the initial synchronization, adopt the full synchronization method to completely transfer all data at the source end to the target end to ensure that the data at the target end is exactly the same as that at the source end; S2. Monitor the change situation of the data in real time through timestamps or database logs to obtain the amount of changed data; S3. During the incremental synchronization process, dynamically select the incremental synchronization or full synchronization method according to the ratio of the amount of changed data to the total amount of data; S4. When the incremental synchronization fails or there is an error in parsing the database file, automatically switch to the full synchronization mode; S5. Perform the full synchronization operation regularly to ensure the integrity and consistency of the data; the period for performing the full synchronization operation regularly is set according to business requirements.

[0039] To more clearly elaborate on the technical solutions of this application, the following will further illustrate through embodiments in specific scenarios.

[0040] This solution aims to achieve efficient, consistent, and resource-optimized data synchronization by dynamically selecting the synchronization method. The core of this solution lies in intelligently switching between full synchronization and incremental synchronization modes based on factors such as the amount of data changes, synchronization frequency, and resource consumption. The implementation process of this solution is as follows: 1. Initial synchronization During the initial synchronization, the full synchronization method is adopted to transfer all the data at the source end to the target end completely, ensuring that the data at the target end is exactly the same as that at the source end.

[0041] 2. Data change monitoring The changes in data are monitored in real time through timestamps (such as the update_time field) or log files (such as the Binlog log of MySQL). The monitoring mechanism can accurately identify data addition, deletion, or modification operations and record the relevant change information, providing a basis for the subsequent selection of the synchronization method.

[0042] To achieve intelligent switching between full synchronization and incremental synchronization, the core lies in automatically selecting the most suitable data synchronization method according to the data change characteristics and business requirements to improve data synchronization efficiency and reduce resource consumption.

[0043] This solution sets up a data change detection module to detect the amount of data changes in this module, thus providing a basis for subsequent intelligent switching.

[0044] In this embodiment, the following two solutions are provided to obtain the changed data of the source table: (1) Timestamp To obtain the data changes of the source table by using timestamps, two fields, namely the creation time (create_time) and the update time (update_time), need to be added to the source table. Among them, the creation time field records the time when the data is first inserted into the source table, and the update time field records the time when the data is updated and modified each time.

[0045] Before each data synchronization, the changed data within a certain time range (such as from the end time of the previous synchronization to the current time) is obtained through the update time field, and the number of rows of the changed data is thus obtained. Its SQL (a database query and programming language used to access data and query, update, and manage relational database systems) query syntax is: SELECT COUNT(*) FROM "yb_table" where update_time>= 'the end time of the previous synchronization' and update_time<'the current time'; Among them, yb_table is the English table name of the source table.

[0046] However, this method has certain drawbacks. For the deleted data, it is impossible to obtain it.

[0047] (2)Database Log Adopt the method of database log to obtain the data changes of the source table. It depends on the log file of the database itself, such as the Binlog file of Mysql. Binlog is used to record the changes to the database inside Mysql in real time, including data operations of addition, deletion, and modification.

[0048] Through some open-source tools, it is very convenient to obtain and parse the changed data in Binlog, and thus obtain the number of rows of the changed data.

[0049] Compared with the timestamp method, adopting the database log method does not require adding corresponding time fields to the source table additionally, and can accurately obtain the deleted data. Therefore, it is more accurate in obtaining the amount of data changes.

[0050] After obtaining the amount of data changes of the source table, it is also necessary to query the total amount of data of the source table at this time. For the total number of rows of the source table, the following standard SQL query syntax can be used for query: SELECT COUNT(*) FROM "yb_table"; Among them, yb_table is the English table name of the source table.

[0051] 3. Dynamic Synchronization Method Selection After completing the first full data synchronization operation, enter the incremental synchronization stage. In this stage, the intervention of intelligent switching is required to balance the efficiency, resource consumption, and data consistency of data synchronization.

[0052] In the incremental synchronization stage, according to the ratio of the monitored amount of changed data to the total amount of data, dynamically select the synchronization method according to the following principles: When the data change ratio is relatively low, preferentially select incremental synchronization to reduce the amount of data transmission and resource consumption; When the data change ratio is relatively high or close to the full amount of data, switch to full data synchronization to avoid the efficiency problem caused by repeated synchronization.

[0053] In this solution, an intelligent switching algorithm module is set up. In this module, according to the amount of changed data and the total amount of data of the source table, calculate the ratio of the changed data, and make a judgment according to the preset conditions, so as to trigger the switch between incremental synchronization and full data synchronization.

[0054] Specifically, as Figure 2 shown, in this embodiment, the implementation steps of the switch between incremental synchronization and full data synchronization are as follows: (1) Calculate the data change ratio In this embodiment, the calculation formula for the data change ratio is: data change ratio = amount of changed data / total amount of data * 100%.

[0055] Assume that the amount of changed data is 150,000 records and the total amount of data is 1,000,000 records. Then, according to the above calculation formula, the obtained data change ratio is 150,000 / 1,000,000 * 100% = 15%.

[0056] (2) Make a judgment according to preset conditions First, different preset thresholds need to be set according to the requirements of different business types.

[0057] If the calculated data change ratio is lower than the preset threshold (such as 10%), it indicates that the amount of changed data is small, and incremental synchronization is selected; If the calculated data change ratio is higher than the preset threshold (such as 10%), it indicates that the amount of changed data is large, and full synchronization is selected.

[0058] For some business scenarios with high real-time requirements, the threshold can be set higher; while for business scenarios with high data consistency requirements, the threshold can be set lower.

[0059] (3) Trigger the switch In the incremental synchronization stage, when the program recognizes that the switching condition is met, the system will pause the incremental synchronization, start a full synchronization, and automatically switch back to the incremental synchronization after the full synchronization operation is completed.

[0060] In addition to the above strategy of dynamically selecting the synchronization method based on the data change ratio (triggering the switch between incremental synchronization and full synchronization), the following key factors of business scenarios can also be comprehensively considered in this solution to dynamically select the appropriate synchronization method: (a) Synchronization frequency For some high-frequency data synchronization scenarios, that is, scenarios with high real-time requirements, incremental synchronization is suitable; For some low-frequency data synchronization scenarios (such as daily synchronization), the appropriate synchronization method can be selected according to the amount of data change.

[0061] (b) Resource situation Select the appropriate data synchronization method according to the network bandwidth and memory resources, etc.

[0062] When the network bandwidth and memory resources are sufficient, the full synchronization method can be selected.

[0063] When the network bandwidth and memory resources are insufficient, it is more resource-saving to select the incremental synchronization method.

[0064] 4. Periodic full - volume synchronization To ensure the integrity and reliability of data, for scenarios with high requirements for data consistency such as finance, based on intelligent switching, a full - volume synchronization operation also needs to be performed regularly (e.g., weekly or monthly). This mechanism can correct data deviations or cumulative errors that may occur during the incremental synchronization process, ensuring the long - term consistency of data between the target end and the source end.

[0065] 5. Fault - tolerance mechanism During the incremental synchronization process, if synchronization fails or database file parsing errors occur, the system will automatically switch to the full - volume synchronization mode to ensure data integrity and consistency. The fault - tolerance mechanism can effectively handle abnormal situations and ensure the reliability of data synchronization.

[0066] Figure 3 Shown is a device for intelligent switching between full - volume synchronization and incremental synchronization proposed in this application. The device includes: An initial synchronization module, used for the first synchronization, adopting the full - volume synchronization method to completely transfer all data from the source end to the target end, ensuring that the data at the target end is exactly the same as that at the source end; A data change detection module, used to monitor the data change situation in real - time through timestamps or database logs and obtain the amount of changed data; An intelligent switching algorithm module, used to dynamically select the incremental synchronization or full - volume synchronization method according to the ratio of the amount of changed data to the total amount of data during the incremental synchronization process; A fault - tolerance module, used to automatically switch to the full - volume synchronization mode when incremental synchronization fails or database file parsing errors occur; A periodic synchronization module, used to perform full - volume synchronization operations regularly to ensure data integrity and consistency.

[0067] When the above - mentioned device runs, it implements the steps of the method for intelligent switching between full - volume synchronization and incremental synchronization disclosed in this application.

[0068] The flowcharts and block diagrams in the accompanying drawings show the possible implementations of devices, methods, and computer program products according to various embodiments of this application, including system architectures, functions, and operations. In these figures, each box can represent a module, a program segment, or a part of code, which contains one or more executable instructions for implementing the specified logical function. It should be noted that each box in the block diagram and / or flowchart, as well as combinations of these boxes, can be implemented using a dedicated hardware - based system to implement the specified function or operation, or can be implemented through a combination of dedicated hardware and computer instructions.

[0069] As Figure 4As shown in the figure, an embodiment of the present application also discloses an electronic device, including: a processor 310, a communication interface 320, a memory 330 for storing computer programs executable by the processor, and a communication bus 340. Among them, the processor 310, the communication interface 320, and the memory 330 complete mutual communication through the communication bus 340. The processor 310 runs the executable computer program to implement the steps of the above-mentioned method for intelligent switching between full synchronization and incremental synchronization.

[0070] It can be understood that in addition to including a memory and a processor, the electronic device may also include an input device (such as a keyboard), an output device (such as a display), and other communication modules. These input devices, output devices, and other communication modules communicate with the processor through an I / O interface (i.e., an input / output interface).

[0071] The operations of the present application can be implemented by writing computer program code using one or more programming languages or combinations thereof. The programming languages include but are not limited to the following types: Object-oriented programming languages, such as Java, Smalltalk, C++, etc.; Conventional procedural programming languages, such as the "C" language or similar programming languages.

[0072] The execution modes of the program code include but are not limited to: Fully executed on the user's computer; Partially executed on the user's computer and partially executed on a remote computer; Executed as an independent software package; Fully executed on a remote computer or server.

[0073] In scenarios involving a remote computer, the remote computer can be connected to the user's computer through any type of network, and the network includes but is not limited to a local area network (LAN) or a wide area network (WAN). In addition, the remote computer can also be connected to an external computer through an Internet service provider, for example, by using the Internet for connection.

[0074] Furthermore, the present application also discloses a computer-readable storage medium. When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device can execute each step of the method for intelligent switching between full synchronization and incremental synchronization disclosed in the present application.

[0075] In the context of the present application, a computer-readable storage medium refers to a tangible medium that can store computer program code and related data. Specific examples include but are not limited to the following: (1) Portable computer disks: Removable magnetic storage media such as floppy disks.

[0076] (2) Hard disks: Fixed storage devices including mechanical hard disks and solid-state drives.

[0077] (3) Random Access Memory (RAM): Volatile storage media for temporarily storing data and program code.

[0078] (4) Read-Only Memory (ROM): Non-volatile storage media for storing fixed programs and data.

[0079] (5) Erasable Programmable Read-Only Memory (EPROM) or Flash Memory: Non-volatile storage media that support multiple erasures and programming.

[0080] (6) Fiber optic storage devices: Storage media based on fiber optic technology.

[0081] (7) Portable Compact Disc Read-Only Memory (CD-ROM): Read-only media for storing data in the form of optical discs.

[0082] (8) Optical storage devices: Storage media based on optical principles such as DVDs and Blu-ray discs.

[0083] (9) Magnetic storage devices: Storage media based on magnetic principles such as magnetic tapes and disks.

[0084] (10) Any suitable combination of the above: For example, combining multiple storage media to meet different storage requirements.

[0085] These computer-readable storage media can be used to store the program code and related data described in this application to support the operation of the program and the persistent storage of data.

[0086] In particular, according to the embodiments of this application, the processes described in the flowcharts can be implemented as computer software programs. For example, the embodiments of this application relate to a computer program product that includes a computer program carried on a non-transitory computer-readable medium. The computer program contains program code for executing the method of intelligent switching between full synchronization and incremental synchronization disclosed in this application. When the computer program is executed by a processing device, it can implement the above functions defined in the embodiments of this application.

[0087] Although the above description contains several specific implementation details, these details should not be construed as limiting the scope of the present application. The above description is only a preferred embodiment of the present application and an explanation of the applied technical principles. Those skilled in the art should understand that the disclosed scope of the present application is not limited to the technical solutions formed by the specific combination of the above technical features. At the same time, the present application should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept.

[0088] Those skilled in the art should also understand that they can modify the technical solutions described in the foregoing embodiments or perform equivalent replacements on some of the technical features without departing from the spirit and scope of the technical solutions of the embodiments of the present application. These modifications or replacements will not cause the essence of the corresponding technical solutions to deviate from the core spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for intelligent switching between full synchronization and incremental synchronization, characterized in that: The method comprises: S1. During the initial synchronization, full synchronization is used to transfer all data from the source end to the target end to ensure that the data on the target end is completely consistent with that on the source end. S2. Monitor data changes in real time through timestamps or database logs to obtain the amount of changed data; S3. During the incremental synchronization process, dynamically select incremental synchronization or full synchronization based on the ratio of the amount of changed data to the total amount of data; S4. When incremental synchronization fails or database file parsing errors occur, it automatically switches to full synchronization mode.

2. The method according to claim 1, characterized in that Step S3 also includes: The ratio of the amount of changed data to the total amount of data is compared with the preset threshold: when the data change ratio is lower than the preset threshold, incremental synchronization is selected; when the data change ratio is higher than the preset threshold, full synchronization is switched.

3. The method according to claim 2, characterized in that The preset threshold is set according to business requirements. A higher threshold is set for business scenarios with high real-time requirements; and a lower threshold is set for business scenarios with high data consistency requirements.

4. The method according to claim 1, characterized in that: The method also includes: S5. Regularly perform full synchronization operations to ensure data integrity and consistency; the period of regular full synchronization operations is set according to business needs.

5. The method according to claim 1, characterized in that The timestamp includes a creation time field and an update time field of the source end table, and the amount of changed data within a certain time range is obtained through the update time field.

6. The method according to claim 1, characterized in that The database log is a Binlog log of MySQL, and the data volume of the addition, deletion and modification operations of the source table is obtained by parsing the Binlog log.

7. The method according to claim 1, characterized in that Step S3 also includes: making a comprehensive judgment based on the synchronization frequency and resource conditions, and dynamically selecting an incremental synchronization or a full synchronization mode; (1) For high-frequency data synchronization scenarios, select incremental synchronization; For low-frequency data synchronization scenarios, select the synchronization method based on the data change ratio; (2) When network bandwidth and memory resources are sufficient, select full synchronization; When network bandwidth and memory resources are insufficient, select incremental synchronization.

8. A device for intelligent switching between full synchronization and incremental synchronization, characterized in that: The device comprises: The initial synchronization module is used to transfer all data from the source end to the target end in full synchronization during the initial synchronization, ensuring that the data on the target end is completely consistent with that on the source end. Data change detection module, used to monitor data changes in real time through timestamps or database logs and obtain the amount of changed data; Intelligent switching algorithm module, used to dynamically select incremental synchronization or full synchronization mode according to the ratio of the amount of changed data to the total amount of data during incremental synchronization; Fault-tolerant module, which is used to automatically switch to full synchronization mode when incremental synchronization fails or database file parsing errors occur; The periodic synchronization module is used to perform full synchronization operations regularly to ensure data integrity and consistency.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for intelligent switching between full synchronization and incremental synchronization as described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: Memory and processor; Memory: used to store computer programs; Processor: used to execute the computer program to implement the steps of the method for intelligent switching between full synchronization and incremental synchronization as described in any one of claims 1-7.

Images