A data synchronization method and device supporting adaptive synchronization mode

Abstract

The present invention discloses a data synchronization method and device supporting an adaptive synchronization mode, wherein the method includes: S1: automatic generation of synchronization configuration information; S2: reading synchronization configuration information; S3: data synchronization preprocessing; S4: data reading processing; S5: data buffering processing; S6: data writing processing; S7: data synchronization post-processing. The present invention automatically determines the synchronization mode through an adaptive decision algorithm, which not only shortens the overall data synchronization time, but also avoids the degradation of service interface request performance during the data synchronization process. The present invention provides a decision algorithm that can automatically decide whether to use incremental synchronization or elegant incremental synchronization for data synchronization. The elegant incremental synchronization method in the present invention achieves no impact and no perception on service requests during the data synchronization process. When an abnormality is found during the synchronization of new data using the solution of the present invention, it can quickly switch to historical data, effectively avoiding the provision of erroneous service data for a long time.

Description

Technical Field

[0001] The present invention belongs to the technical field of data integration, and in particular relates to a data synchronization method and device supporting an adaptive synchronization mode. Background Art

[0002] In the era of digital economy, data has become an important asset for enterprises. Enterprises need to establish a complete data system to better enhance the value of data assets. The construction of the application data layer ADS (Application Data Service) is a key link in enhancing the value of data. Data needs to be synchronized to different storage media according to different usage scenarios to meet the business needs for different throughput and response time. For search and query service scenarios, the existing technology generally uses periodic synchronization of data in the data warehouse to the application data layer ADS with ES (ElasticSearch) as the storage medium. However, in actual production practice, the following problems will be encountered:

[0003] (1) If full synchronization is used, the overall synchronization cycle will be extended, and full synchronization is required even when only a small amount of data is updated, resulting in unnecessary waste of resources.

[0004] (2) If the incremental synchronization method is used fixedly, then when the amount of incremental updated data is large, during the data synchronization process, it will cause service request response delays, degrade service interface request performance, and the user experience will not be friendly enough. Summary of the invention

[0005] To solve the above problems, the present invention discloses a data synchronization method and device that supports adaptive synchronization mode, which automatically determines the synchronization mode through an adaptive decision algorithm, thereby shortening the overall data synchronization time and avoiding the degradation of service interface request performance during data synchronization.

[0006] To achieve the above object, the technical solution of the present invention is as follows:

[0007] A data synchronization method supporting an adaptive synchronization mode comprises the following steps:

[0008] S1: Synchronous configuration information is automatically generated;

[0009] Generate corresponding full synchronization configuration information or incremental synchronization configuration information according to the data storage status of the source data table and the target data table;

[0010] S2: read synchronization configuration information;

[0011] Read the generated full synchronization configuration information or incremental synchronization configuration information content as a string, and parse the string into a JSON object;

[0012] S3: data synchronization preprocessing;

[0013] When step S2 reads the full synchronization configuration information, the full synchronization preprocessing process is entered; when step S2 reads the incremental synchronization configuration information, an adaptive decision algorithm is used to determine whether to enter the incremental synchronization preprocessing process or the elegant incremental synchronization preprocessing process;

[0014] S4: data reading and processing;

[0015] Read data in batches from the source database and proceed to the subsequent steps for data buffering;

[0016] S5: data buffer processing;

[0017] When receiving read data, temporarily store the data in the first cache. When the first cache reaches the maximum capacity, push the data in the first cache to the data pipeline and clear the current first cache. When receiving a data pull instruction, store the data in the data pipeline into the second cache so that subsequent steps can obtain data from the second cache.

[0018] S6: data writing process;

[0019] Pull data from the second cache, assemble the data and write it in batches to the destination database;

[0020] S7: data synchronization post-processing;

[0021] After data synchronization is completed, post-processing including data preheating and resource recovery is performed.

[0022] Furthermore, the step S1 specifically includes the following sub-steps:

[0023] S11: Configure source connection, destination connection and tolerance time information;

[0024] S12: Read the maximum time partition max_dt of the source data table;

[0025] S13: Determine whether the maximum time partition max_dt is equal to the null value NULL. If the value of max_dt is equal to the null value NULL, it means that no data has been generated yet. At this time, data synchronization is not required and the process ends directly. If the value of max_dt is not equal to the null value NULL, proceed to step S14.

[0026] S14: read the maximum time max_time of the destination data table;

[0027] S15: Determine whether the maximum time max_time is equal to the null value NULL. If the value of max_time is equal to the null value NULL, proceed to step S16; if the value of max_time is not equal to the null value NULL, proceed to step S17;

[0028] S16: Set the time partition field equal to max_dt, and automatically generate full synchronization configuration information based on the configuration parameters;

[0029] S17: Compare max_dt and max_time. If max_dt is less than or equal to max_time, it means that incremental data has not been generated yet and data synchronization is not required at this time. If max_dt is greater than max_time, proceed to step S18.

[0030] S18: Set the time partition field to be greater than max_time, and automatically generate incremental synchronization configuration information in combination with configuration parameters.

[0031] Furthermore, in step S11, the source connection information includes the source database type, IP address, port number, user name, password, instance name, source data table name and custom SQL for data retrieval; the destination connection information includes the ES connection address, port number, user name, password and number of indexes retained; the tolerance time is the performance delay time tolerated by the business request interface.

[0032] Furthermore, the specific process of automatically generating synchronization configuration information is as follows: if it is full synchronization, the where condition of the source-side custom data retrieval SQL is that the maximum time partition field is equal to max_dt; if it is incremental synchronization, the where condition of the source-side custom data retrieval SQL is that the maximum time partition field is greater than max_time; based on the source-side connection information, the index name, alias, and mapping configuration of the target ES database are automatically generated; combined with the configuration template, the full or incremental synchronization configuration information is automatically generated.

[0033] Furthermore, the step S3 specifically includes the following process:

[0034] S31: Check configuration parameters;

[0035] S32: Determine whether full synchronization is performed, and proceed to step S33 if full synchronization is not performed, and proceed to step S35 if full synchronization is performed;

[0036] S33: Get the current ES index name, read the average data synchronization rate speed from the cache, execute the adaptive decision algorithm df(), and go to step S34 when the adaptive decision algorithm df() decision result indicates that incremental synchronization is adopted; and go to step S35 when the adaptive decision algorithm df() decision result indicates that elegant incremental synchronization is adopted;

[0037] S34: Update the index name in the configuration information to the current index name, and proceed to step S36;

[0038] S35: Create an ES index and an ES mapping; when elegant incremental synchronization is used, a reindex operation is also performed; proceed to step S36;

[0039] S36: Shard data based on the table primary key.

[0040] Furthermore, the adaptive decision algorithm df() formula is as follows:

[0041]

[0042] Where n represents the total number of data to be synchronized, range() represents the decision range, tt represents the configured tolerance time, speed represents the average data synchronization rate, and factor represents the compensation coefficient, which mainly compensates for the speed reduction caused by resource usage conflicts during data synchronization. When n is within the decision range, the decision result is 1, indicating that incremental synchronization is adopted. When n is not within the decision range, the decision result is 0, indicating that elegant incremental synchronization is adopted.

[0043] Furthermore, the step S7 specifically includes the following process:

[0044] S71: Set the ES index alias; first obtain the index name set under the specified ES alias. If the index name is not equal to the configured new index name, remove the alias of the index, add the alias to the new index, and preheat the data through the first query;

[0045] S72: clean up historical indexes; use the alias to perform fuzzy query and sort by creation time to obtain the current index set. If the number exceeds the number of reserved indexes, clean up historical indexes with earlier creation time.

[0046] S73: Resource recovery processing; data reading module database connection and data writing module ES client destruction processing;

[0047] S74: Synchronization performance statistics: statistics on the start and end time of synchronization, calculation of the total time consumption and synchronization rate, and statistics on the number of successes and failures;

[0048] S75: The synchronization rate is stored in the cache; if the synchronization rate in the current cache is empty or less than the new synchronization rate, the new synchronization rate value is stored in the cache.

[0049] Furthermore, when the latest written business data is found to be incorrect, it is possible to quickly switch to the historical index and quickly restore the business.

[0050] A data synchronization device supporting an adaptive synchronization mode, used to implement the above-mentioned data synchronization method supporting an adaptive synchronization mode, comprising: a configuration generation module, a configuration reading module, a pre-processing module, a data reading module, a data buffering module, a data writing module and a post-processing module;

[0051] The configuration generation module is used to generate corresponding full synchronization configuration information or incremental synchronization configuration information according to the data storage status of the source data table and the target data table;

[0052] The configuration reading module is used to read the generated full synchronization configuration information or incremental synchronization configuration information into the memory;

[0053] The preprocessing module is used to verify whether the configuration items are correct, create ES indexes and mappings in advance, and use the full synchronization method or the adaptive decision algorithm to determine the incremental synchronization method or the elegant incremental synchronization method;

[0054] The data reading module is used to read data in batches from the source database and send the data to the data buffer module;

[0055] The data buffer module is used to connect the data reading module and the data writing module, serve as a data transmission channel between the two, and cache synchronized data;

[0056] The data writing module is used to continuously consume data from the data buffer module and write the data in batches to the destination database;

[0057] The post-processing module is used for post-processing after data synchronization is completed.

[0058] Furthermore, the post-processing includes: ES index alias switching, old index cleaning, data reading module database connection and data writing module ES client destruction processing, synchronization process performance indicator statistics, synchronization rate storage in cache and other processing.

[0059] The beneficial effects of the present invention are:

[0060] 1. The solution provided by the present invention implements data synchronization based on an adaptive synchronization method, shortens the overall data synchronization time, and avoids the degradation of service interface request performance during the data synchronization process.

[0061] 2. The solution of the present invention provides a decision algorithm that can automatically decide whether to use incremental synchronization or elegant incremental synchronization to synchronize data.

[0062] 3. The elegant incremental synchronization method in the solution of the present invention achieves no impact or perception on business requests during the data synchronization process.

[0063] 4. When an abnormality is found during the synchronization of new data using the solution of the present invention, it is possible to quickly switch to historical data, effectively avoiding the provision of erroneous business data for a long time. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 A flow chart of a data synchronization method supporting an adaptive synchronization mode provided by the present invention.

[0065] Figure 2 Automatically generate a flowchart of steps for synchronizing configuration information.

[0066] Figure 3 Flowchart of preprocessing steps for data synchronization.

[0067] Figure 4 This is the data synchronization post-processing flow chart.

[0068] Figure 5 A schematic diagram of the structure of a data synchronization device supporting an adaptive synchronization method provided by the present invention. DETAILED DESCRIPTION

[0069] The technical solution provided by the present invention will be described in detail below in conjunction with specific embodiments. It should be understood that the following specific implementation methods are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0070] An embodiment of the present invention provides a data synchronization method supporting an adaptive synchronization mode, and the process is as follows: Figure 1 As shown, the specific steps include:

[0071] S1: Synchronous configuration information is automatically generated

[0072] The processing flow of this step is as follows Figure 2 As shown, specifically including:

[0073] S11: Configure the source connection, destination connection, and tolerance time. The source connection information includes the source database type, IP address, port number, user name, password, instance name, source data table name, and custom SQL for data retrieval. The destination connection information includes the ES connection address, port number, user name, password, and number of indexes to be retained. The tolerance time refers to the performance delay time that the business request interface tolerates. The specific configuration value can be determined comprehensively based on the specific business request cache expiration time. The default value is 300 seconds.

[0074] S12: Read the maximum time partition max_dt of the source data table.

[0075] S13: Determine whether the maximum time partition max_dt is equal to the null value NULL. If the value of max_dt is equal to the null value NULL, it means that data has not been generated yet. At this time, data synchronization is not required and the process ends directly; if the value of max_dt is not equal to the null value NULL, enter step S14.

[0076] S14: Read the maximum time max_time of the destination data table.

[0077] S15: Determine whether the maximum time max_time is equal to the null value NULL. If the value of max_time is equal to the null value NULL, proceed to step S16; if the value of max_time is not equal to the null value NULL, proceed to step S17.

[0078] S16: Set the time partition field equal to max_dt, and automatically generate full synchronization configuration information in combination with configuration parameters. The specific process is as follows: the where condition of the source-side custom data acquisition SQL is that the maximum time partition field is equal to max_dt; based on the source-side connection information, automatically generate the index name, alias, and mapping configuration of the target ES database; combined with the configuration template, automatically generate full synchronization configuration information.

[0079] S17: Compare max_dt and max_time. If max_dt is less than or equal to max_time, it means that incremental data has not been generated yet and data synchronization is not required at this time; if max_dt is greater than max_time, go to step S18.

[0080] S18: Set the time partition field to be greater than max_time, and automatically generate incremental synchronization configuration information in combination with configuration parameters. The specific process is: the where condition of the source-side custom data acquisition SQL is that the maximum time partition field is greater than max_time; based on the source-side connection information, automatically generate the index name, alias, and mapping configuration of the target ES database; combined with the configuration template, automatically generate incremental synchronization configuration information.

[0081] S2: Read synchronization configuration information

[0082] This step reads the generated full synchronization configuration information or incremental synchronization configuration information content as a string and parses the string into a JSON object.

[0083] S3: Data synchronization preprocessing

[0084] The processing flow of this step is as follows Figure 3As shown in the figure, there are three different preprocessing processes: full synchronization preprocessing process, incremental synchronization preprocessing process and elegant incremental synchronization preprocessing process. When S2 reads the full synchronization configuration information, it enters the full synchronization preprocessing process. When S2 reads the incremental synchronization configuration information, it uses an adaptive decision algorithm to determine the incremental synchronization preprocessing method or the elegant incremental synchronization preprocessing method.

[0085] The full synchronization preprocessing process mainly includes configuration parameter checking, creating ES indexes, creating ES mappings, and sharding data according to the table primary key. The configuration parameter check mainly verifies the configuration parameters in the synchronization configuration information, such as the batch read quantity cannot be less than 1, the user name and password cannot be empty, etc. Different types of read modules and write modules each implement specific parameter configuration check items. Creating an ES index mainly uses the destination ES connection information and index name configured in the configuration information, and sends the create index interface to the ES cluster. Creating an ES mapping mainly uses the mapping information configured in the configuration information, and sends the create ES mapping interface to the ES cluster. The data sharding process according to the table primary key includes first using the primary key field configured in the configuration information to obtain the minimum and maximum value interval range, and then equally divides it into N sub-intervals. N represents the number of tasks that need to read data concurrently. The default value is 5, which can be configured according to the actual scenario. Finally, it is assembled into a data retrieval SQL condition to facilitate subsequent concurrent data reading from the source database, thereby improving data synchronization performance.

[0086] The incremental synchronization preprocessing process mainly includes configuration parameter checking, obtaining the current ES index name, reading the average data synchronization speed from the cache, executing the adaptive decision algorithm df(), updating the index name in the configuration information to the current index name, and sharding the data according to the table primary key. To obtain the current ES index name, you can use the alias to send a fuzzy query request to the ES cluster to obtain the index name pointed to by the current alias for use in subsequent process processing.

[0087] The mentioned adaptive decision algorithm df() formula is as follows:

[0088]

[0089] Among them, n represents the total number of data to be synchronized, range() represents the decision range, tt represents the configured tolerance time, speed represents the average data synchronization rate, and factor represents a coefficient randomly generated between [0.1 and 0.25]. This coefficient mainly considers the speed reduction caused by resource usage conflicts during data synchronization to compensate. When n is within the range, the decision result is 1, indicating that incremental synchronization is adopted; when n is not within the range, the decision result is 0, indicating that elegant incremental synchronization is adopted.

[0090] The elegant incremental synchronization preprocessing process mainly includes configuration parameter checking, obtaining the current ES index name, reading the average data synchronization speed, executing the adaptive decision algorithm df(), creating ES indexes, creating ES mappings, executing reindex operations, and sharding data according to the table primary key. To execute the reindex operation, use the previously obtained index name and the index name in the configuration information, send a reindex request to the ES cluster, and migrate the data under the current index to the new index. The reindex request can set sharding parameters to increase parallelism.

[0091] S4: Data reading and processing

[0092] The data reading module reads data in batches from the source database and sends the data to the data buffer module.

[0093] S5: Data buffering

[0094] The data buffer module receives the data instruction sent by the data reading module and temporarily stores the data in the first cache. When the first cache reaches the maximum capacity, the data in the cache is pushed to the data pipeline and the current first cache is cleared. The data buffer module receives the data pull instruction from the data writing module and stores the data in the data pipeline into the second cache. The data writing module then obtains the data from the second cache. The data buffer module is used to connect the data reading module and the data writing module as the data transmission channel between the two, cache the synchronized data, and thus improve the overall synchronization efficiency.

[0095] S6: Data writing process

[0096] Pull data from the second cache of the data buffer module, assemble the data and write them in batches to the destination database.

[0097] S7: Data synchronization post-processing.

[0098] The processing flow of this step is as follows Figure 4 As shown, including:

[0099] S71: Set the ES index alias. First, obtain the index name set under the specified ES alias. If the index name is not equal to the configured new index name, remove the alias of the index, add the alias to the new index, and preheat the data through the first query to avoid a large delay in the first data query after the index switch.

[0100] S72: Clean up historical indexes. Use aliases for fuzzy query and sort by creation time to obtain the current index set. If the number exceeds the number of retained indexes (the default value is 2), clean up historical indexes with earlier creation times. When the latest written business data is found to be incorrect, you can quickly switch to the historical index to quickly restore the business.

[0101] S73: Resource recovery processing. Destroy the database connection resources (of any type, including oracle, mysql, greenplum, etc.) used by the data reading module and the ES client resources used by the data writing module.

[0102] S74: Synchronization performance statistics. The specific processing includes: counting the start and end time of data synchronization, calculating the total time consumption and synchronization rate, and counting the number of successes and failures.

[0103] S75: Storing the synchronization rate in the cache. The specific processing includes: if the synchronization rate in the current cache is empty or less than the new synchronization rate, storing the new synchronization rate value in the cache.

[0104] The embodiment of the present invention further provides a data synchronization device supporting an adaptive synchronization mode, which is used to implement the data synchronization method supporting an adaptive synchronization mode. Figure 5 As shown, it specifically includes: a configuration generation module, a configuration reading module, a pre-processing module, a data reading module, a data buffer module, a data writing module and a post-processing module.

[0105] The configuration generation module mainly generates corresponding full synchronization configuration information or incremental synchronization configuration information according to the data storage status of the source data table and the target data table, and specifically implements the content of step S1 in the data synchronization method supporting the adaptive synchronization mode.

[0106] The configuration reading module is mainly used to read the generated full synchronization configuration information or incremental synchronization configuration information into the memory, and specifically implements the content of step S2 in the data synchronization method supporting the adaptive synchronization mode.

[0107] The preprocessing module is mainly used to verify whether the configuration items are correct, read the synchronization rate from the cache, create ES indexes and mappings in advance, use the full synchronization method, or use an adaptive decision algorithm to determine the incremental synchronization method or the elegant incremental synchronization method. Specifically implement the content of step S3 in the data synchronization method that supports the adaptive synchronization method.

[0108] The data reading module is mainly used to read data in batches from the source database and send the data to the data buffer module. Specifically implement the content of step S4 in the data synchronization method supporting the adaptive synchronization mode.

[0109] The data buffer module is mainly used to connect the data reading module and the data writing module as a data transmission channel between the two, and cache the synchronized data, thereby improving the overall synchronization efficiency. Specifically implement the content of step S5 in the data synchronization method supporting the adaptive synchronization mode.

[0110] The data writing module is mainly used to continuously consume data from the data buffer module and write the data in batches to the destination database, specifically implementing the content of step S6 in the data synchronization method supporting the adaptive synchronization mode.

[0111] The post-processing module is mainly used to perform necessary post-processing after data synchronization is completed, including ES index alias switching, old index cleaning, data reading module database connection and data writing module ES client destruction processing, synchronization process performance indicator statistics, synchronization rate storage in cache, etc. It specifically implements the content of step S7 in the data synchronization method that supports adaptive synchronization.

[0112] It should be noted that the above content only illustrates the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. For ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications all fall within the protection scope of the claims of the present invention.

Claims

1. A data synchronization method supporting an adaptive synchronization mode, characterized in that: The steps include: S1: Synchronous configuration information is automatically generated; Generate corresponding full synchronization configuration information or incremental synchronization configuration information according to the data storage status of the source data table and the target data table. The specific steps include the following: S11: Configure source connection, destination connection and tolerance time information; S12: Read the maximum time partition max_dt of the source data table; S13: Determine whether the maximum time partition max_dt is equal to the null value NULL. If the value of max_dt is equal to the null value NULL, it means that no data has been generated yet. At this time, data synchronization is not required and the process ends directly. If the value of max_dt is not equal to the null value NULL, proceed to step S14; S14: read the maximum time max_time of the destination data table; S15: Determine whether the maximum time max_time is equal to the null value NULL. If the value of max_time is equal to the null value NULL, proceed to step S16; if the value of max_time is not equal to the null value NULL, proceed to step S17; S16: Set the time partition field to be equal to max_dt, set the where condition of the source-side custom data fetching SQL to be that the maximum time partition field is equal to max_dt, and automatically generate full synchronization configuration information based on the configuration parameters; S17: Compare max_dt and max_time. If max_dt is less than or equal to max_time, it means that incremental data has not been generated yet and data synchronization is not required at this time. If max_dt is greater than max_time, proceed to step S18. S18: Set the time partition field to be greater than max_time, set the where condition of the source-side custom data retrieval SQL to be greater than max_time, and automatically generate incremental synchronization configuration information based on the configuration parameters; S2: read synchronization configuration information; Read the generated full synchronization configuration information or incremental synchronization configuration information content as a string, and parse the string into a JSON object; S3: data synchronization preprocessing; When step S2 reads the full synchronization configuration information, the full synchronization preprocessing process is entered; when step S2 reads the incremental synchronization configuration information, an adaptive decision algorithm is used to determine whether to enter the incremental synchronization preprocessing process or the elegant incremental synchronization preprocessing process; specifically, the following sub-steps are included: S31: Check configuration parameters; S32: Determine whether full synchronization is performed, and proceed to step S33 if full synchronization is not performed, and proceed to step S35 if full synchronization is performed; S33: Get the current ES index name, read the average data synchronization rate speed from the cache, execute the adaptive decision algorithm df(), and go to step S34 when the adaptive decision algorithm df() decision result indicates that incremental synchronization is adopted; and go to step S35 when the adaptive decision algorithm df() decision result indicates that elegant incremental synchronization is adopted; The adaptive decision algorithm df() formula is as follows: range0=(0,tt·speed.(1+factor)] Among them, n represents the total number of data to be synchronized, range() represents the decision range, tt represents the configured tolerance time, speed represents the average data synchronization rate, and factor represents the compensation coefficient, which mainly considers the speed reduction caused by resource usage conflicts during data synchronization and compensates for it. When n is within the decision range, the decision result is 1, indicating that incremental synchronization is adopted; when n is not within the decision range, the decision result is 0, indicating that elegant incremental synchronization is adopted. S34: Update the index name in the configuration information to the current index name, and proceed to step S36; S35: Create an ES index and an ES mapping; when elegant incremental synchronization is used, a reindex operation is also performed; proceed to step S36; S36: Shard data according to the table primary key; S4: data reading and processing; Read data in batches from the source database and proceed to the subsequent steps for data buffering; S5: data buffer processing; When receiving read data, temporarily store the data in the first cache. When the first cache reaches the maximum capacity, push the data in the first cache to the data pipeline and clear the current first cache. When receiving a data pull instruction, store the data in the data pipeline into the second cache so that subsequent steps can obtain data from the second cache. S6: data writing process; Pull data from the second cache, assemble the data and write it in batches to the destination database; S7: data synchronization post-processing; After data synchronization is completed, post-processing including data preheating and resource recovery is performed.

2. The data synchronization method supporting adaptive synchronization according to claim 1, characterized in that: In step S11, the source connection information includes the source database type, IP address, port number, user name, password, instance name, source data table name and custom SQL for data retrieval; the destination connection information includes the ES connection address, port number, user name, password and number of indexes reserved; the tolerance time is the performance delay time tolerated by the business request interface.

3. The data synchronization method supporting adaptive synchronization according to claim 1, characterized in that: The specific process of automatically generating synchronization configuration information is as follows: If it is full synchronization, the where condition of the source-side custom data retrieval SQL is that the maximum time partition field is equal to max_dt; If it is incremental synchronization, the where condition of the source-side custom data retrieval SQL is that the maximum time partition field is greater than max_time; based on the source-side connection information, the index name, alias, and mapping configuration of the target ES database are automatically generated; combined with the configuration template, the full or incremental synchronization configuration information is automatically generated.

4. The data synchronization method supporting adaptive synchronization according to claim 1, characterized in that: The step S7 specifically includes the following process: S71: Set the ES index alias; first obtain the index name set under the specified ES alias. If the index name is not equal to the configured new index name, remove the alias of the index, add the alias to the new index, and preheat the data through the first query; S72: clean up historical indexes; use the alias to perform fuzzy query and sort by creation time to obtain the current index set. If the number exceeds the number of reserved indexes, clean up historical indexes with earlier creation time. S73: Resource recovery processing; Destruction processing of database connection of data reading module and ES client of data writing module; S74: Synchronization performance statistics: statistics on the start and end time of synchronization, calculation of the total time consumption and synchronization rate, and statistics on the number of successes and failures; S75: The synchronization rate is stored in the cache; If the synchronization rate in the current cache is empty or less than the new synchronization rate, the new synchronization rate value is stored in the cache.

5. The data synchronization method supporting adaptive synchronization according to claim 4, characterized in that: When the latest business data written is found to be incorrect, it is possible to quickly switch to the historical index and quickly restore the business.

6. A data synchronization device supporting an adaptive synchronization method, characterized in that: A data synchronization method supporting an adaptive synchronization mode for implementing any one of claims 1 to 5, comprising: a configuration generation module, a configuration reading module, a preprocessing module, a data reading module, a data buffering module, a data writing module and a post-processing module; The configuration generation module is used to generate corresponding full synchronization configuration information or incremental synchronization configuration information according to the data storage status of the source data table and the target data table; The configuration reading module is used to read the generated full synchronization configuration information or incremental synchronization configuration information into the memory; The preprocessing module is used to verify whether the configuration items are correct, create ES indexes and mappings in advance, and use the full synchronization method or the adaptive decision algorithm to determine the incremental synchronization method or the elegant incremental synchronization method; The data reading module is used to read data in batches from the source database and send the data to the data buffer module; The data buffer module is used to connect the data reading module and the data writing module, serve as a data transmission channel between the two, and cache synchronized data; The data writing module is used to continuously consume data from the data buffer module and write the data in batches to the destination database; The post-processing module is used for post-processing after data synchronization is completed.

7. The data synchronization device supporting adaptive synchronization according to claim 6, characterized in that: The post-processing includes: ES index alias switching, old index cleaning, data reading module database connection and data writing module ES client destruction processing, synchronization process performance indicator statistics, and synchronization rate cache processing.

Images