Incremental data synchronization method and system based on incremental field adaptive intelligent matching

Abstract

The application provides a kind of incremental data synchronization method and system based on incremental field adaptive intelligent matching, belongs to incremental data synchronization technical field, the method: configure incremental synchronization task, receive the incremental synchronization mode selected by user;According to the incremental synchronization mode selected by user, execute target table query SQL, obtain incremental reference value;According to incremental reference value, generate source table query SQL and target table preprocessing SQL;According to preset synchronization time, execute incremental synchronization task through middleware, query incremental data from source data table through source table query SQL, and delete duplicate data in target table through target table preprocessing SQL, then update the incremental data queried from source data table to target table.The application can automatically identify incremental field type or receive user-defined conditions, adapt to various synchronization scenarios, dynamically generate SQL statements, realize intelligent synchronization, execute tasks according to preset time with the help of middleware, reduce manual intervention and improve efficiency.

Description

Technical Field

[0001] This application belongs to the field of incremental data synchronization technology, specifically relating to an incremental data synchronization method and system based on adaptive intelligent matching of incremental fields. Background Technology

[0002] With the development of information technology, data in enterprise information systems is growing rapidly. To achieve data consistency between different systems, data synchronization has become a key technology. However, traditional incremental data synchronization methods have many limitations, specifically as follows:

[0003] First, traditional incremental data synchronization methods rely on a single type of incremental field (such as an auto-incrementing ID or a fixed timestamp), making it difficult to adapt to diverse field scenarios such as numeric and date / time types. This is especially true in scenarios with fields lacking a clear incrementing pattern, where it's difficult to effectively identify incremental changes. Second, traditional incremental data synchronization methods require manual configuration of incremental fields and synchronization rules. This necessitates repeated configuration whenever the data table structure changes or new business tables are added, resulting in extremely low efficiency. Furthermore, manual configuration is prone to errors due to misjudgment of field types, increasing the risk of data synchronization omissions or errors. Third, in high-concurrency scenarios, traditional incremental synchronization methods use fixed polling intervals, making it difficult to balance real-time performance with system load balancing. For example, while short-cycle polling improves real-time performance, it may generate dirty data if the target table is not updated in a timely manner. Finally, traditional incremental data synchronization methods are often designed for specific business scenarios and lack general support for multiple types of incremental fields (e.g., automatic time zone conversion during cross-event time field synchronization), making it difficult to adapt to complex business needs. Summary of the Invention

[0004] In a first aspect, embodiments of this application provide an incremental data synchronization method based on adaptive intelligent matching of incremental fields, comprising the following steps:

[0005] S1. Configure the incremental synchronization task and receive the incremental synchronization method selected by the user;

[0006] If the selected incremental synchronization method is incremental field, the incremental field type will be automatically identified, and the preset rule template will be loaded to generate the target table query SQL.

[0007] If the selected incremental synchronization method is incremental condition, it receives the user-defined variable name and the target table query SQL.

[0008] S2. Execute the target table query SQL according to the incremental synchronization method selected by the user to obtain the incremental baseline value;

[0009] S3. Generate source table query SQL and target table preprocessing SQL based on the incremental baseline value;

[0010] S4. The middleware executes incremental synchronization tasks according to the preset synchronization time. It queries incremental data from the source data table using source table query SQL, deletes duplicate data from the target table using target table preprocessing SQL, and then updates the target table with the incremental data queried from the source data table.

[0011] Furthermore, the specific steps of step S1 are as follows:

[0012] S11. Pre-set rule templates for each incremental field type; the incremental field types include numeric fields, date / time fields, and string fields;

[0013] S12. Respond to the user's incremental synchronization task configuration through the data synchronization application, and receive the incremental synchronization method selected by the user;

[0014] When the user selects incremental field as the incremental synchronization method, proceed to step S13;

[0015] When the user selects incremental synchronization method as incremental condition, proceed to step S14;

[0016] S13. Match the incremental field type of the user-selected incremental field with the metadata of the source data table database, obtain the rule template of the matching incremental field type, generate the target table query SQL, and proceed to step S2.

[0017] S14. Receive the user-defined variable name, and then receive the target table query SQL defined by the user for the variable name.

[0018] Furthermore, in step S11, a rule template for querying the maximum value is pre-configured for numeric fields, a rule template for querying the latest timestamp is configured for date and time fields, and a lexicographical increment rule template is configured for string fields.

[0019] Furthermore, the specific steps of step S13 are as follows:

[0020] Obtain the incremental field selected by the user, and identify the data type of the incremental field through the metadata in the source data table's database;

[0021] If the data type is INT, BIGINT, or NUMBER, it will be recognized as a numeric field.

[0022] If the data type is DATE or TIMESTAMP, it will be recognized as a date / time field.

[0023] If the data type is VARCHAR or TEXT, it will be recognized as a string field.

[0024] Furthermore, the specific steps of step S2 are as follows:

[0025] S21. Determine the incremental synchronization method selected by the user;

[0026] If it is an incremental field, proceed to step S22;

[0027] If it is an incremental condition, proceed to step S26;

[0028] S22. Determine the data type of the incremental field;

[0029] If it is a numeric field, proceed to step S23;

[0030] If it is a date / time field, proceed to step S24;

[0031] If it is a string field, proceed to step S25;

[0032] S23. Execute the SQL query for the target table of the rule template that queries the maximum value, and return the query result as the parameter variable of the maximum value. Set the returned maximum value as the incremental baseline value, and proceed to step S3.

[0033] S24. Execute the SQL query for the target table of the rule template that queries the latest timestamp, and return the query result as a parameter variable of the latest timestamp. Set the returned latest timestamp as the incremental baseline value, and proceed to step S3.

[0034] S25. Execute the target table query SQL of the lexicographical incremental rule template, and return the query results through the parameter variable of the maximum value after sorting by lexicographical order. Set the returned maximum value after sorting by lexicographical order as the incremental baseline value, and proceed to step S3.

[0035] S26. Execute the user-defined target table query SQL, inject the query results into the user-defined variable name, and set the variable value corresponding to the user-defined variable name to the incremental baseline value.

[0036] Furthermore, the specific steps of step S3 are as follows:

[0037] S31. Identify the incremental baseline value;

[0038] If it is the maximum value, proceed to step S32;

[0039] If it is the latest timestamp, proceed to step S33;

[0040] If it is the maximum value after sorting by dictionary, proceed to step S35;

[0041] If the variable name corresponds to the variable value defined by the user, proceed to step S36;

[0042] S32. Using the maximum value as the input parameter, with the goal of obtaining incremental data in numeric fields that are greater than the maximum value, construct the source table query SQL and the target table preprocessing SQL and proceed to step S37.

[0043] S33. Identify the time span from the date and time field entered by the user;

[0044] S34. Determine the start and end query times based on the latest timestamp and time span, and construct the source table query SQL and target table preprocessing SQL with the goal of obtaining incremental data between the start and end query times. Proceed to step S37.

[0045] S35. Using the maximum value after dictionary sorting as the input parameter, and aiming to obtain incremental data of string fields that are greater than the maximum value after dictionary sorting, construct source table query SQL and target table preprocessing SQL, and proceed to step S37;

[0046] S36. Using the variable values ​​corresponding to the user-defined variable names as input parameters, construct the source table query SQL and the target table preprocessing SQL according to the user-preset conditional expressions;

[0047] S37. Encapsulate the constructed source table query SQL and target table preprocessing SQL into an incremental synchronization task as an atomic transaction task and provide it to the middleware, and record the incremental baseline value.

[0048] Furthermore, the specific steps of step S4 are as follows:

[0049] S41. Respond to the user-configured incremental synchronization task execution time and trigger task execution according to the incremental synchronization task execution time;

[0050] S42. Generate an incremental synchronization task instance and submit the incremental synchronization task instance to the middleware;

[0051] S43. The middleware executes preprocessing SQL on the target table and deletes duplicate data in the target table;

[0052] S44. The middleware executes source table query SQL on the source data table to obtain incremental data from the source data table;

[0053] S45. Synchronize the retrieved incremental data to the target table.

[0054] Secondly, embodiments of this application also provide an incremental data synchronization system based on adaptive intelligent matching of incremental fields, comprising:

[0055] The incremental task configuration module is used to configure incremental synchronization tasks and receives the incremental synchronization method selected by the user.

[0056] If the selected incremental synchronization method is incremental field, the incremental field type will be automatically identified, and the preset rule template will be loaded to generate the target table query SQL.

[0057] If the selected incremental synchronization method is incremental condition, it receives the user-defined variable name and the target table query SQL.

[0058] The incremental baseline value determination module is used to execute target table query SQL according to the incremental synchronization method selected by the user to obtain the incremental baseline value;

[0059] The dynamic SQL generation module is used to generate source table query SQL and target table preprocessing SQL based on incremental baseline values;

[0060] The incremental task execution module is used to execute incremental synchronization tasks according to a preset synchronization time through middleware. It queries incremental data from the source data table using source table query SQL, deletes duplicate data from the target table using target table preprocessing SQL, and then updates the target table with the incremental data queried from the source data table.

[0061] Thirdly, embodiments of this application also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the incremental data synchronization method based on incremental field adaptive intelligent matching as described in the first aspect.

[0062] Fourthly, embodiments of this application also provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the incremental data synchronization method based on adaptive intelligent matching of incremental fields as described in the first aspect.

[0063] As can be seen from the above technical solutions, this application has the following advantages:

[0064] The incremental data synchronization method and system based on adaptive intelligent matching of incremental fields provided in this application can dynamically identify the incremental field type and generate corresponding SQL statements through adaptive intelligent matching, reducing manual intervention and improving synchronization efficiency and accuracy. At the same time, it supports multiple field types and complex business scenarios, enhancing flexibility and scalability. It can effectively solve the problems of field type limitations, complexity of manual configuration, contradiction between real-time performance and accuracy, and insufficient flexibility and scalability in traditional incremental data synchronization methods. Attached Figure Description

[0065] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart illustrating the incremental data synchronization method based on adaptive intelligent matching of incremental fields according to the present invention.

[0067] Figure 2 This is a schematic diagram of the incremental data synchronization system based on adaptive intelligent matching of incremental fields according to the present invention. Detailed Implementation

[0068] The various embodiments of this disclosure will be described more fully in the detailed steps of the incremental data synchronization method based on adaptive intelligent matching of incremental fields below. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0069] For example, with the rapid development of information technology, the amount of data in enterprise information systems is experiencing explosive growth. To ensure data consistency between different systems, data synchronization technology has become a key means to achieve this goal. However, existing traditional incremental data synchronization methods have revealed many limitations in practical applications, specifically as follows:

[0070] First, traditional incremental data synchronization methods mainly rely on a single type of incremental field (such as an auto-incrementing primary key ID or a fixed timestamp field). This design is difficult to adapt to complex scenarios that include multiple types of fields, such as numeric and date / time fields. Especially when dealing with fields that do not have a clear incrementing pattern, traditional methods cannot effectively identify incremental changes, thus limiting their applicability in diverse business scenarios.

[0071] Secondly, traditional incremental data synchronization methods require manual specification of incremental fields and synchronization rules. When the data table structure changes or a new business table is added, the configuration needs to be repeated, which is not only inefficient, but also prone to synchronization omissions or errors due to human error (such as misjudging field types), thus increasing the risk of data synchronization failures.

[0072] Furthermore, in high-concurrency scenarios, traditional incremental synchronization methods typically employ fixed polling intervals to detect data changes. However, this approach struggles to balance real-time performance with system load. For instance, while short-cycle polling can improve the real-time performance of data synchronization, it may lead to the generation of dirty data if the target table has not yet been updated, thus affecting data accuracy and consistency.

[0073] Finally, traditional incremental data synchronization methods are mostly designed for specific business scenarios and lack general support for multiple types of incremental fields. For example, when synchronizing time fields across time zones, they cannot automatically handle time zone conversions. These limitations make it difficult for traditional methods to meet increasingly complex business needs, especially in cross-system and cross-domain data interaction scenarios.

[0074] In conclusion, traditional incremental data synchronization methods are no longer sufficient to meet the demands for efficient, accurate, and flexible data synchronization in the complex and ever-changing data environments of modern enterprise information systems. Therefore, there is an urgent need for a new incremental data synchronization technology that can overcome these limitations and offer greater adaptability and flexibility.

[0075] To address the aforementioned issues, this embodiment provides an incremental data synchronization method based on adaptive intelligent matching of incremental fields. By automatically identifying the incremental field type or receiving user-defined incremental conditions, it flexibly adapts to various synchronization scenarios; it dynamically generates SQL statements based on incremental baseline values ​​to achieve intelligent synchronization; it uses middleware to execute tasks at preset times, reducing manual intervention and improving efficiency; and it uses pre-processed SQL to delete duplicate data, ensuring synchronization accuracy.

[0076] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0077] Please see Figure 1 The diagram shows a flowchart of an incremental data synchronization method based on adaptive intelligent matching of incremental fields in a specific embodiment. The method includes the following steps:

[0078] S1. Configure the incremental synchronization task and receive the incremental synchronization method selected by the user;

[0079] If the selected incremental synchronization method is incremental field, the incremental field type will be automatically identified, and the preset rule template will be loaded to generate the target table query SQL.

[0080] If the selected incremental synchronization method is incremental condition, it receives the user-defined variable name and the target table query SQL.

[0081] It should be noted that allowing users to select incremental fields or incremental conditions provides a flexible configuration method; automatically identifying the incremental field type and loading rule templates reduces the complexity of manual configuration; and supporting multiple field types through preset rule templates improves versatility and scalability.

[0082] S2. Execute the target table query SQL according to the incremental synchronization method selected by the user to obtain the incremental baseline value;

[0083] It should be noted that the system dynamically executes the target table query SQL based on the incremental synchronization method selected by the user to obtain the incremental baseline value; by executing the target table query SQL, the accuracy of the obtained incremental baseline value is ensured; and user-defined variable names and query SQL are supported to enhance flexibility.

[0084] S3. Generate source table query SQL and target table preprocessing SQL based on the incremental baseline value;

[0085] It should be noted that the source table query SQL and target table preprocessing SQL are intelligently constructed based on the type of incremental baseline value, which improves the intelligence level of the synchronization task; it ensures that the source table query SQL can accurately query incremental data and the target table preprocessing SQL can accurately delete duplicate data.

[0086] S4. The middleware executes incremental synchronization tasks according to the preset synchronization time. It queries incremental data from the source data table using source table query SQL, deletes duplicate data from the target table using target table preprocessing SQL, and then updates the target table with the incremental data queried from the source data table.

[0087] It should be noted that middleware executes tasks at preset times, improving scheduling flexibility; middleware manages task instances, ensuring efficient task execution; and preprocessed SQL removes duplicate data from the target table, ensuring accurate data synchronization.

[0088] This embodiment can flexibly adapt to various incremental synchronization scenarios by automatically identifying incremental field types or receiving user-defined incremental conditions; it dynamically generates source table query SQL and target table preprocessing SQL based on incremental baseline values ​​to achieve intelligent incremental data synchronization; it executes tasks at preset times through middleware, reducing manual intervention and improving synchronization efficiency; and it deletes duplicate data in the target table through preprocessing SQL to ensure the accuracy of data synchronization.

[0089] Furthermore, as a refinement and extension of the specific implementation of the above embodiments, in order to fully illustrate the specific implementation process in this embodiment, another incremental data synchronization method based on adaptive intelligent matching of incremental fields is provided. This method includes the following steps:

[0090] S1. Configure the incremental synchronization task and receive the incremental synchronization method selected by the user;

[0091] If the selected incremental synchronization method is incremental field, the incremental field type will be automatically identified, and the preset rule template will be loaded to generate the target table query SQL; the specific steps of step S1 are as follows:

[0092] S11. Pre-set rule templates for each incremental field type in the data synchronization application; the incremental field types include numeric fields, date / time fields, and string fields;

[0093] In step S11, a rule template for querying the maximum value is pre-configured for numeric fields, a rule template for querying the latest timestamp is pre-configured for date and time fields, and a lexicographical increment rule template is pre-configured for string fields.

[0094] For example, a rule template for querying the maximum value:

[0095] case "NUMERIC":

[0096] return "WHERE ${field}>${max_value}";

[0097] Where ${max_value} is the numeric incremental field input by the user, and ${field} is the maximum value of the returned incremental field.

[0098] Rule template for querying the latest timestamp:

[0099] case "DATETIME":

[0100] return "WHERE ${field}>${last_max_time};

[0101] Where ${last_max_time} is the date and time field entered by the user, and ${field} is the latest timestamp of the returned incremental field;

[0102] Rule template for lexicographical increment rule template:

[0103] case "STRING":

[0104] return "WHERE ${field}>'${last_string}' ORDER BY ${field} ASC";

[0105] Where ${field} is the string-type incremental field selected by the user, and ${last_string} is the maximum value of this field in the returned target table after sorting by lexicographical order;

[0106] It should be noted that by configuring targeted rule templates for different types of incremental fields, it ensures that appropriate SQL statements can be generated for each field type; it supports multiple field types such as numeric, date / time, and string, enhancing its versatility.

[0107] S12. Respond to the user's incremental synchronization task configuration through the data synchronization application, and receive the incremental synchronization method selected by the user;

[0108] When the user selects incremental field as the incremental synchronization method, proceed to step S13;

[0109] When the user selects incremental synchronization method as incremental condition, proceed to step S14;

[0110] S13. Match the incremental field type of the user-selected incremental field with the metadata of the source data table database, obtain the rule template of the matching incremental field type, generate the target table query SQL, and proceed to step S2.

[0111] The specific steps of step S13 are as follows:

[0112] Obtain the incremental field selected by the user, and identify the data type of the incremental field through the metadata in the source data table's database;

[0113] If the data type is INT, BIGINT, or NUMBER, it will be recognized as a numeric field.

[0114] If the data type is DATE or TIMESTAMP, it will be recognized as a date / time field.

[0115] If the data type is VARCHAR or TEXT, it will be recognized as a string field.

[0116] It should be noted that by accurately identifying the data type of incremental fields through database metadata, the accuracy of field type identification is improved; and by automatically matching rule templates to generate SQL statements, the workload of manual configuration is reduced.

[0117] S14. Receive the user-defined variable name, and then receive the target table query SQL defined by the user for the variable name;

[0118] If the selected incremental synchronization method is incremental condition, it receives the user-defined variable name and the target table query SQL.

[0119] It should be noted that by setting rule templates for different types of incremental fields, the configuration process is simplified and scalability is improved; users can select incremental fields or incremental conditions to meet different business needs; and by matching incremental field types with metadata, the accurate identification of field types is ensured, reducing human error.

[0120] S2. Execute the target table query SQL according to the incremental synchronization method selected by the user to obtain the incremental baseline value; the specific steps of step S2 are as follows:

[0121] S21. Determine the incremental synchronization method selected by the user;

[0122] If it is an incremental field, proceed to step S22;

[0123] If it is an incremental condition, proceed to step S26;

[0124] S22. Determine the data type of the incremental field;

[0125] If it is a numeric field, proceed to step S23;

[0126] If it is a date / time field, proceed to step S24;

[0127] If it is a string field, proceed to step S25;

[0128] S23. Execute the SQL query for the target table of the rule template that queries the maximum value, and return the query result as the parameter variable of the maximum value. Set the returned maximum value as the incremental baseline value, and proceed to step S3.

[0129] S24. Execute the SQL query for the target table of the rule template that queries the latest timestamp, and return the query result as a parameter variable of the latest timestamp. Set the returned latest timestamp as the incremental baseline value, and proceed to step S3.

[0130] S25. Execute the target table query SQL of the lexicographical incremental rule template, and return the query results through the parameter variable of the maximum value after sorting by lexicographical order. Set the returned maximum value after sorting by lexicographical order as the incremental baseline value, and proceed to step S3.

[0131] S26. Execute the user-defined target table query SQL, inject the query results into the user-defined variable name, and set the variable value corresponding to the user-defined variable name to the incremental baseline value;

[0132] It should be noted that the target table query SQL is dynamically executed based on the data type of the incremental field to obtain the incremental baseline value, ensuring the dynamism of the synchronization task; user-defined variable names and query SQL are supported, enhancing flexibility.

[0133] S3. Generate source table query SQL and target table preprocessing SQL based on the incremental baseline value;

[0134] It should be noted that if the task is triggered multiple times, or if it is interrupted and re-executed during the task execution, some data may be inserted into the target table repeatedly. In high-concurrency scenarios, the target table may be modified by other operations during the execution of the synchronous task, resulting in data inconsistency. By executing the target table preprocessing SQL, the duplicate data that may exist in the target table can be deleted, which can ensure that the data in the target table is consistent with the incremental data in the source table.

[0135] The specific steps of step S3 are as follows:

[0136] S31. Identify the incremental baseline value;

[0137] If it is the maximum value, proceed to step S32;

[0138] If it is the latest timestamp, proceed to step S33;

[0139] If it is the maximum value after sorting by dictionary, proceed to step S35;

[0140] If the variable name corresponds to the variable value defined by the user, proceed to step S36;

[0141] S32. Using the maximum value as the input parameter, with the goal of obtaining incremental data in numeric fields that are greater than the maximum value, construct the source table query SQL and the target table preprocessing SQL and proceed to step S37.

[0142] For example, the source table query SQL is:

[0143] select * from source_table

[0144] where value > ${max_value};

[0145] The target table preprocessing SQL is:

[0146] delete * from target_table

[0147] where value > ${max_value};

[0148] S33. Identify the time span from the date and time field entered by the user;

[0149] S34. Determine the start and end query times based on the latest timestamp and time span, and construct the source table query SQL and target table preprocessing SQL with the goal of obtaining incremental data between the start and end query times. Proceed to step S37.

[0150] For example, the source table query SQL is:

[0151] select * from source_table

[0152] where createTime between “start_time” and “end_time”;

[0153] The target table preprocessing SQL is:

[0154] delect * from source_table

[0155] where createTime between “start_time” and “end_time”;

[0156] It should be noted that for date and time types, the precision can be set as year, month, day, hour, minute, and second. The time span, scheduling execution time, and the time before and after can be set. For example, the design precision is month, with 2 before and 3 after.

[0157] If the scheduling time (denoted as ${systemTime}) is "2025-05-08 14:00:30" during execution, and the field selected is createTime, then the WHERE clause can be set to the form between...and..., then:

[0158] The SQL query for the source table is: select * from source_table

[0159] where createTime between "2025-03-08 14:00:30" and "2025-08-08 14:00:30";

[0160] The preprocessing SQL for the target table is: delete * from target_table

[0161] where createTime between "2025-03-08 14:00:30" and "2025-08-08 14:00:30";

[0162] S35. Using the maximum value after dictionary sorting as the input parameter, and aiming to obtain incremental data of string fields that are greater than the maximum value after dictionary sorting, construct source table query SQL and target table preprocessing SQL, and proceed to step S37;

[0163] For example, the source table query SQL is:

[0164] SELECT * FROM source_table

[0165] WHERE ${field}>'${last_string};

[0166] The target table preprocessing SQL is:

[0167] DELETE FROM target_table

[0168] WHERE ${field}>'${last_string}';

[0169] S36. Using the variable values ​​corresponding to the user-defined variable names as input parameters, construct the source table query SQL and the target table preprocessing SQL according to the user-preset conditional expressions;

[0170] S37. Encapsulate the constructed source table query SQL and target table preprocessing SQL into an incremental synchronization task as an atomic transaction task and provide it to the middleware, and record the incremental baseline value.

[0171] It should be noted that the source table query SQL and target table preprocessing SQL are intelligently constructed based on the type of incremental baseline value, which improves the intelligence level of the synchronization task; the synchronization task is encapsulated into atomic transaction tasks to ensure the integrity and consistency of data synchronization.

[0172] S4. The middleware executes incremental synchronization tasks according to a preset synchronization time. It queries incremental data from the source table using source table query SQL, deletes duplicate data from the target table using target table preprocessing SQL, and then updates the target table with the incremental data retrieved from the source table. The specific steps of step S4 are as follows:

[0173] S41. The scheduling application responds to the incremental synchronization task execution time configured by the user, and triggers task execution according to the incremental synchronization task execution time;

[0174] S42. Generate an incremental synchronization task instance and submit the incremental synchronization task instance to the middleware;

[0175] S43. The middleware executes preprocessing SQL on the target table and deletes duplicate data in the target table;

[0176] S44. The middleware executes source table query SQL on the source data table to obtain incremental data from the source data table;

[0177] S45. Synchronize the retrieved incremental data to the target table;

[0178] It should be noted that by flexibly configuring the execution time of synchronization tasks through the scheduling application platform, scheduling flexibility is improved; and by managing task instances through middleware, efficient task execution and accurate data synchronization are ensured.

[0179] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0180] like Figure 2 As shown, the following are embodiments of the incremental data synchronization system based on adaptive intelligent matching of incremental fields provided in this disclosure. This system and the incremental data synchronization method based on adaptive intelligent matching of incremental fields in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the incremental data synchronization system based on adaptive intelligent matching of incremental fields, please refer to the embodiments of the incremental data synchronization method based on adaptive intelligent matching of incremental fields described above.

[0181] The system includes:

[0182] The incremental task configuration module is used to configure incremental synchronization tasks and receives the incremental synchronization method selected by the user.

[0183] If the selected incremental synchronization method is incremental field, the incremental field type will be automatically identified, and the preset rule template will be loaded to generate the target table query SQL.

[0184] If the selected incremental synchronization method is incremental condition, it receives the user-defined variable name and the target table query SQL.

[0185] The incremental baseline value determination module is used to execute target table query SQL according to the incremental synchronization method selected by the user to obtain the incremental baseline value;

[0186] The dynamic SQL generation module is used to generate source table query SQL and target table preprocessing SQL based on incremental baseline values;

[0187] The incremental task execution module is used to execute incremental synchronization tasks according to a preset synchronization time through middleware. It queries incremental data from the source data table using source table query SQL, deletes duplicate data from the target table using target table preprocessing SQL, and then updates the target table with the incremental data queried from the source data table.

[0188] This embodiment achieves the integrity and coordination of task configuration, baseline value determination, SQL generation, and task execution through the interactive collaboration of the incremental task configuration module, incremental baseline value determination module, dynamic SQL generation module, and incremental task execution module.

[0189] The incremental data synchronization method based on adaptive intelligent matching of incremental fields provided in this application can be applied to electronic devices. Those skilled in the art will understand that the electronic device structure involved in the embodiments of this invention does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the electronic device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.

[0190] Electronic devices may include processors, external memory interfaces, internal memory, universal serial bus (USB) interfaces, charging management modules, power management modules, batteries, wireless communication modules, audio modules, speakers, microphones, sensor modules, buttons, cameras, displays, and SIM card interfaces, etc.

[0191] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0192] A processor may include one or more processing units, such as a central processing unit (CPU), an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0193] The processor can serve as the nerve center and command center of an electronic device. The controller can generate operation control signals based on the instruction opcode and timing signals to control the fetching and execution of instructions.

[0194] The processor may also include memory for storing instructions and data. In some embodiments, the memory in the processor is a cache memory. This memory can store instructions or data that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from this memory. This avoids repeated accesses, reduces processor latency, and thus improves system efficiency.

[0195] The aforementioned electronic device implements the incremental data synchronization method based on adaptive intelligent matching of incremental fields as described in this application, configuring incremental synchronization tasks. It receives the incremental synchronization method selected by the user; if the selected incremental synchronization method is incremental field, it automatically identifies the incremental field type and loads a preset rule template to generate a target table query SQL; if the selected incremental synchronization method is incremental condition, it receives the user-defined variable name and the target table query SQL; it executes the target table query SQL according to the user-selected incremental synchronization method to obtain the incremental baseline value; it generates the source table query SQL and the target table preprocessing SQL based on the incremental baseline value; it executes the incremental synchronization task according to a preset synchronization time through middleware, queries incremental data from the source data table using the source table query SQL, and deletes duplicate data from the target table using the target table preprocessing SQL, and then updates the target table with the incremental data queried from the source data table. This achieves the beneficial effects of automatically identifying the incremental field type or receiving user-defined incremental conditions to flexibly adapt to various synchronization scenarios; dynamically generating SQL statements based on the incremental baseline value to achieve intelligent synchronization; reducing manual intervention and improving efficiency by executing tasks according to a preset time using middleware; and ensuring synchronization accuracy by deleting duplicate data using preprocessing SQL.

[0196] The storage medium provided in this application stores a program product capable of implementing an incremental data synchronization method based on adaptive intelligent matching of incremental fields.

[0197] The incremental data synchronization method based on adaptive intelligent matching of incremental fields includes: configuring an incremental synchronization task and receiving the incremental synchronization method selected by the user; if the selected incremental synchronization method is incremental field, automatically identifying the incremental field type and loading a preset rule template to generate the target table query SQL; if the selected incremental synchronization method is incremental condition, receiving the user-defined variable name and the target table query SQL; executing the target table query SQL according to the incremental synchronization method selected by the user to obtain the incremental baseline value; generating the source table query SQL and the target table preprocessing SQL according to the incremental baseline value; executing the incremental synchronization task according to the preset synchronization time through middleware, querying incremental data from the source data table through the source table query SQL, deleting duplicate data in the target table through the target table preprocessing SQL, and then updating the target table with the incremental data queried from the source data table.

[0198] In some possible implementations, the incremental data synchronization method based on adaptive intelligent matching of incremental fields disclosed herein can be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0199] The storage medium disclosed herein may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0200] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An incremental data synchronization method based on adaptive intelligent matching of incremental fields, characterized in that, Includes the following steps: S1. Configure the incremental synchronization task and receive the incremental synchronization method selected by the user; If the selected incremental synchronization method is incremental field, the incremental field type will be automatically identified, and the preset rule template will be loaded to generate the target table query SQL. If the selected incremental synchronization method is incremental condition, it receives the user-defined variable name and the target table query SQL. S11. Pre-set rule templates for each incremental field type; the incremental field types include numeric fields, date / time fields, and string fields; Pre-configure rule templates for querying the maximum value for numeric fields, rule templates for querying the latest timestamp for date and time fields, and lexicographical increment rule templates for string fields; S2. Execute the target table query SQL according to the incremental synchronization method selected by the user to obtain the incremental baseline value; the specific steps of step S2 are as follows: S21. Determine the incremental synchronization method selected by the user; If it is an incremental field, proceed to step S22; If it is an incremental condition, proceed to step S26; S22. Determine the data type of the incremental field; If it is a numeric field, proceed to step S23; If it is a date / time field, proceed to step S24; If it is a string field, proceed to step S25; S23. Execute the SQL query for the target table of the rule template that queries the maximum value, and return the query result as the parameter variable of the maximum value. Set the returned maximum value as the incremental baseline value, and proceed to step S3. S24. Execute the SQL query for the target table of the rule template that queries the latest timestamp, and return the query result as a parameter variable of the latest timestamp. Set the returned latest timestamp as the incremental baseline value, and proceed to step S3. S25. Execute the target table query SQL of the lexicographical incremental rule template, and return the query result through the parameter variable of the maximum value after sorting by lexicographical order. Set the returned maximum value after sorting by lexicographical order as the incremental base value, and proceed to step S3. S26. Execute the user-defined target table query SQL, inject the query results into the user-defined variable name, and set the variable value corresponding to the user-defined variable name to the incremental baseline value; S3. Generate source table query SQL and target table preprocessing SQL based on the incremental baseline value; S4. The middleware executes incremental synchronization tasks according to the preset synchronization time. It queries incremental data from the source data table using source table query SQL, deletes duplicate data from the target table using target table preprocessing SQL, and then updates the target table with the incremental data queried from the source data table.

2. The incremental data synchronization method based on adaptive intelligent matching of incremental fields according to claim 1, characterized in that, Step S1 also includes the following specific steps: S12. Respond to the user's incremental synchronization task configuration through the data synchronization application, and receive the incremental synchronization method selected by the user; When the user selects incremental field as the incremental synchronization method, proceed to step S13; When the user selects incremental synchronization method as incremental condition, proceed to step S14; S13. Match the incremental field type of the user-selected incremental field with the metadata of the source data table database, obtain the rule template of the matching incremental field type, generate the target table query SQL, and proceed to step S2. S14. Receive the user-defined variable name, and then receive the target table query SQL defined by the user for the variable name.

3. The incremental data synchronization method based on adaptive intelligent matching of incremental fields according to claim 2, characterized in that, The specific steps of step S13 are as follows: Obtain the incremental field selected by the user, and identify the data type of the incremental field through the metadata in the source data table's database; If the data type is INT, BIGINT, or NUMBER, it will be recognized as a numeric field. If the data type is DATE or TIMESTAMP, it will be recognized as a date / time field. If the data type is VARCHAR or TEXT, it will be recognized as a string field.

4. The incremental data synchronization method based on adaptive intelligent matching of incremental fields according to claim 3, characterized in that, The specific steps of step S3 are as follows: S31. Identify the incremental baseline value; If it is the maximum value, proceed to step S32; If it is the latest timestamp, proceed to step S33; If it is the maximum value after sorting by dictionary, proceed to step S35; If the variable name corresponds to the variable value defined by the user, proceed to step S36; S32. Using the maximum value as the input parameter, with the goal of obtaining incremental data in numeric fields that are greater than the maximum value, construct the source table query SQL and the target table preprocessing SQL and proceed to step S37. S33. Identify the time span from the date and time field entered by the user; S34. Determine the start and end query times based on the latest timestamp and time span, and construct the source table query SQL and target table preprocessing SQL with the goal of obtaining incremental data between the start and end query times. Proceed to step S37. S35. Using the maximum value after dictionary sorting as the input parameter, and aiming to obtain incremental data of string fields that are greater than the maximum value after dictionary sorting, construct source table query SQL and target table preprocessing SQL, and proceed to step S37; S36. Using the variable values ​​corresponding to the user-defined variable names as input parameters, construct the source table query SQL and the target table preprocessing SQL according to the user-preset conditional expressions; S37. Encapsulate the constructed source table query SQL and target table preprocessing SQL into an incremental synchronization task as an atomic transaction task and provide it to the middleware, and record the incremental baseline value.

5. The incremental data synchronization method based on adaptive intelligent matching of incremental fields according to claim 4, characterized in that, The specific steps of step S4 are as follows: S41. Respond to the user-configured incremental synchronization task execution time and trigger task execution according to the incremental synchronization task execution time; S42. Generate an incremental synchronization task instance and submit the incremental synchronization task instance to the middleware; S43. The middleware executes preprocessing SQL on the target table and deletes duplicate data in the target table; S44. The middleware executes source table query SQL on the source data table to obtain incremental data from the source data table; S45. Synchronize the retrieved incremental data to the target table.

6. An incremental data synchronization system based on adaptive intelligent matching of incremental fields, employing the incremental data synchronization method based on adaptive intelligent matching of incremental fields as described in any one of claims 1-5, characterized in that, include: The incremental task configuration module is used to configure incremental synchronization tasks and receives the incremental synchronization method selected by the user. If the selected incremental synchronization method is incremental field, the incremental field type will be automatically identified, and the preset rule template will be loaded to generate the target table query SQL. If the selected incremental synchronization method is incremental condition, it receives the user-defined variable name and the target table query SQL. The incremental baseline value determination module is used to execute target table query SQL according to the incremental synchronization method selected by the user to obtain the incremental baseline value; The dynamic SQL generation module is used to generate source table query SQL and target table preprocessing SQL based on incremental baseline values; The incremental task execution module is used to execute incremental synchronization tasks according to a preset synchronization time through middleware. It queries incremental data from the source data table using source table query SQL, deletes duplicate data from the target table using target table preprocessing SQL, and then updates the target table with the incremental data queried from the source data table.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the incremental data synchronization method based on incremental field adaptive intelligent matching as described in any one of claims 1 to 5.

8. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the incremental data synchronization method based on incremental field adaptive intelligent matching as described in any one of claims 1 to 5.

Images