[0029] In order to further illustrate the techniques and efficacy of the present invention to achieve predetermined purposes, the present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments.
[0030] The steps of the method flow in the specification in the present invention and the steps of flowchart flowcharts in the drawings of the present invention must not be strictly implemented by step labels, and the method steps can change the execution order. Moreover, some steps can be omitted, combine multiple steps into one step, and / or decompose one step into multiple steps.
[0031] like figure 1 As shown, a database synchronization method based on an incremental field according to an embodiment of the present invention, including:
[0032] S100, select the preset data field from the source table as the increment field, set the first endpoint value, and the second end point value, where the incremental field has an incremental relationship, and the first end point value is less than the second end point value;
[0033] For example, an initial value can be provided as a first end point value, or in the case where the first initial value is not set, the numerical increment field can be used as the first end point value, the date type increment field can be The current time date is used as the first end point value, and the timestamp type increment field can be used as the first end point value at the current time time. The second endpoint value default value can be a run time of the synchronization task set by the user, and the user can set a delay time, then the actual task runtime should be: set the runtime + delay time.
[0034] For example, when the data is required to be synchronized in phases, the synchronization operation can be divided into multiple stages according to the data to be synchronized, wherein the second end point value for termination is set for each phase. It should be noted that the first end point value of the next phase may be the second end point value of the last synchronization of the program automatically recorded, and a good connection when different phase data is synchronized.
[0035] S200, screening data between the first endpoint and the second endpoint as increment data from the incremental field, and synchronizing the increment data into the target table.
[0036] According to some embodiments of the invention, the synchronization method also includes:
[0037] Set the preset cycle, automatically obtain increment data from the source table according to the preset cycle.
[0038] For example, the present invention can set an automatic synchronization time, for example, can set the synchronization operation of the database automatically starting the database daily 00:00. When the daily 00:00 time is reached, the incremental data to be synchronized is collected from the database according to the first endpoint value and the second end point value. It will be appreciated that the above time is merely illustrative, and the present application can set any time according to the user needs.
[0039] In some embodiments of the invention, when all of the incremental data is completed in the current cycle, the startup of the next preset cycle is stopped, and the synchronization operation of the current cycle increment data is continued.
[0040] As described above, if the next 00:00 is reached, the new data synchronization operation can be stopped when the data is completed in the next day, and the new data synchronization operation can be stopped, and the data synchronization operation last day is continued.
[0041] According to some embodiments of the present invention, the received data synchronously starts the configuration time, and the incremental data is automatically obtained from the source table when the data synchronously starts the configuration time.
[0042] That is, the user can set the startup time of automatic synchronization, as described above, the user can set an extension start time (such as 5) on the basis of the original set synchronization data operation time (such as 00:00). Minute, the actual startup time of this task is 00:05, thereby avoiding data synchronization at 00:00 at 00:00, may result in incremental data from 00:00, missing synchronization problem.
[0043] In some embodiments of the invention, the data type of the incremental field is a numeric, time date type, and timestamp type. It should be noted that the database synchronization method proposed based on incremental data can be implemented, and an incremental acquisition and synchronization update of any data type with increment attributes in the database can be implemented.
[0044] A database synchronization device 100 based on an incremental field according to an embodiment of the present invention, includes: a configuration module 10 and an incremental synchronizing module 20.
[0045] The configuration module 10 is used to select the preset data field from the source table as the increment field, set the first end point value, and the second end point value, where the incremental field has an incremental relationship, and the first end point value is less than the second end point value. ;
[0046] For example, the initial value can be set as the first end point value by the configuration module 10, or when the first initial value is not set, the numerical increment field can be used as the first end point value, the date type increase The quantity field can be used as the first end point value at the current time date, and the timestamp type increment field can be used as the first end point value at the current time time. The second endpoint value default value can be a run time of the synchronization task set by the user, and the user can set a delay time, then the actual task runtime should be: set the runtime + delay time.
[0047] For example, when the data is required to be synchronized in phases, the synchronization operation can be divided into multiple stages according to the data to be synchronized, wherein the second end point value for termination is set for each phase. It should be noted that the first end point value of the next phase may be the second end point value of the last synchronization of the program automatically recorded, and a good connection when different phase data is synchronized.
[0048] The incremental synchronization module 20 is configured to screen data between the first endpoint and the second end point as incremental data from the incremental field, and synchronize the increment data to the target table.
[0049] According to some embodiments of the present invention, the configuration module 10 is also used to set a preset cycle, and the incremental data is automatically acquired from the source table according to the preset cycle.
[0050] For example, an automatic synchronization time can be set by configuring module 10, for example, the synchronization operation of the database can be activated automatically at 00:00 daily. When the daily 00:00 time is reached, the incremental data to be synchronized is collected from the database according to the first endpoint value and the second end point value. It will be appreciated that the above time is merely illustrative, and the present application can set any time according to the user needs.
[0051] In some embodiments of the invention, the synchronizing apparatus 100 further includes: a scheduling module 30 for stopping the startup of the next preset cycle when all of the current cycle is completed, stop starting the next preset cycle, and continues to complete the current cycle. Synchronization of incremental data.
[0052] As described above, if the next 00:00 is reached, the new data synchronization operation can be stopped when the data is completed in the next day, and the new data synchronization operation can be stopped, and the data synchronization operation last day is continued.
[0053] According to some embodiments of the present invention, the configuration module 10 is also used to set the data synchronization start configuration time, and the incremental data is automatically obtained from the source table when the data synchronously starts the configuration time.
[0054] That is, the user can set the startup time of automatic synchronization, as described above, the user can set an extension start time (such as 5) on the basis of the original set synchronization data operation time (such as 00:00). Minute, the actual startup time of this task is 00:05, thereby avoiding data synchronization at 00:00 at 00:00, may result in incremental data from 00:00, missing synchronization problem.
[0055] In some embodiments of the invention, the data type of the incremental field is a numeric, time date type, and timestamp type. It should be noted that the database synchronization method proposed based on the incremental field of the present invention can implement incremental acquisition and synchronization update of any data type with increment attributes in the database.
[0056] In summary, the database synchronization method and synchronization device 100 proposed by the present invention have the following advantages:
[0057] The user can specify the increment field and the initial value by visualization configuration, the initial value, support numeric, timeday type, and timestamp type. By the incremental field and the initial value, the data initialization script and the increment script are automatically constructed, and the configuration period task implements increment data acquisition. Compared to the existing incremental acquisition method, other components need to be configured, complex, high error rate, high maintenance cost, the solution of the present invention is simple, easy to operate, easy to use, and can be popular.
[0058] The database synchronization method and the synchronization device 100 based on the increment field according to the present invention will be described in detail below with reference to the accompanying drawings. It is understood that the following description is merely exemplary description, and is not to be understood to be specifically limited to the present invention.
[0059] The present invention is to solve how to make more convenient, more efficient, more accurate and automated in a web system, reduce data initialization and data incremental acquisitions between heterogeneous data sources, and reduce artificial intervention.
[0060] The present invention is based on a conventional ETL timestamp mode. By setting the initial value by specifying the increment field when the integrated task is configured, the incremental field is smaller than the content of the initialized content, which is greater than and equal to the initial value. The data is used as incremental data to achieve data initialization and data increment acquisition between heterogeneous data sources.
[0061] With the present invention, it is not necessary to achieve data initialization and data incremental acquisition between automated heterogeneous data sources by means of other excess data acquisition components, thereby reducing manual operation, enhancing data acquisition efficiency and performance.
[0062] 1, task configuration process:
[0063] When configuring tasks, first you need to specify an increment field, the increment field is a field in the source end table, and the field type can be numeric or date type.
[0064] After specifying the increment field, you need to set the initial value for the corresponding increment field. If the initial value is not set, the system will default to the initial value, the value is 0, the date type is the current time.
[0065] 2, the detailed process of the task run:
[0066] The whole process is described in detail below with a specific task, and the task details are as follows:
[0067] Data source [0002] MySQL table, hypothesis table name mysql_src Incremental field [0004] UpdateTime [0005] Incremental field type [0006] Date type
[0068] 1) Construct data initialization query conditions;
[0069] If it is the first running task, the SQL template is as follows according to the increment field and the initial value constructor:
[0070] Select * from mysql_src where updatetime
[0071] 2) Construct data incremental query conditions;
[0072] If it is the first incremental run, according to the increment field and the initial value, add the current runtime constructor SQL, the SQL template is as follows:
[0073] = $ {initial_value} andUpdatetime
[0074] If it is not the first incremental run, according to the increment field and the last run time, plus the current runtime constructor SQL, the SQL template is as follows:
[0075] = $ {executionTime_1} andUpdatetime
[0076] After querying SQL construction, you can read data in the source end table according to the constructed SQL statement.
[0077] 3) Timing time for running time
[0078] In order to ensure that the query SQL constructed before the task is accurate, it is necessary to strictly control the timing of the record running task time (termination value of the data filter range), the scheme is completed after the current data synchronization operation is completed, then record this task runtime As the initial value of data screening as the next data synchronization. If the data synchronization task failed, the task runtime will not be written back, then the next task will re-construct a new query SQL.
[0079] 3) The cycle schedule guarantees incremental data on time;
[0080] The present invention can adopt a configuration cycle scheduling strategy, according to the cycle scheduling strategy, the setup task is established, no excess manual operation, mitigation of artificial work, and realize automated data collection.
[0081] The database synchronization method and synchronization device 100 proposed by the present invention have the following advantages:
[0082] Configuration: Specify the increment field and the initial value by visualization configuration, and support numerical type and date type.
[0083] Automation: The data initialization script and increment script are automatically constructed by increment fields and initial values, and the configuration period task implements increment data acquisition.
[0084] Simple: At present, many incremental acquisition methods need to configure other components, complex, high error rate, high maintenance cost, this document is simple, easy to operate, easy to use, and popular.
[0085] By way of the embodiments of the specific embodiments, the techniques and efficacy of the present invention can be made more in-depth and detailed, however, reference only to provide reference and description, are not used to use the present invention. limit.
