Data synchronization method and system, and computer readable storage medium

Abstract

The present disclosure relates to a data synchronization method and system, and a computer-readable storage medium. The method includes: in response to an operation of configuring a data source in a first interactive interface, connecting multiple data sources; then, in response to an operation of creating a task in a second interactive interface, generating a task configuration file; thereafter, in response to an operation of starting a data synchronization task in a third interactive interface, synchronizing the data to be synchronized in the source data source to the target data source. In this way, in this embodiment, data transmission between different data sources can be achieved through visual configuration operations without repeated development; and the configuration process is simple, which can reduce the learning cost.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data synchronization method and system, and a computer-readable storage medium. Background Art

[0002] At present, different industries form different data centers, and each data center is isolated from each other. When data from two data centers is required, the data from one data center can be synchronized to the other data center. For example, the data ETL (Extract-Transform-Load) conversion tool in the relevant technology can extract data from the cluster of one data center and synchronize it to the cluster of another data center. For this reason, the relevant technology usually needs to develop corresponding ETL conversion tools for each data center. However, repeated development of ETL conversion tools for different data centers or different projects will waste development time, extend the development cycle, and reduce development efficiency. Summary of the invention

[0003] The present disclosure provides a data synchronization method and system, and a computer-readable storage medium to address the deficiencies of related technologies.

[0004] According to a first aspect of an embodiment of the present disclosure, a data synchronization method is provided, including:

[0005] In response to an operation of configuring a data source in the first interactive interface, connecting a plurality of data sources;

[0006] In response to an operation of creating a task in the second interactive interface, generating a task configuration file;

[0007] In response to the operation of starting the data synchronization task in the third interactive interface, the data to be synchronized in the source data source is synchronized to the target data source.

[0008] Optionally, in response to an operation of configuring a data source in the first interactive interface, connecting multiple data sources includes:

[0009] In response to an operation of selecting a data source in the first interactive interface, obtaining a data source corresponding to the operation; the data source includes a source data source and a target data source;

[0010] In response to an operation of modifying a configuration parameter in the first interactive interface, acquiring a target configuration parameter;

[0011] In response to an operation of testing the connection between the source data source and the target data source in the first interactive interface, attempting to connect the source data source and the target data source;

[0012] In response to successful connection, the data source is saved.

[0013] Optionally, the synchronization rule adopts hot update, including:

[0014] In response to an operation indicating modification of a target configuration parameter, the target configuration parameter of a configuration table in a task configuration file is modified, and the task configuration file is stored in a target database.

[0015] Optionally, in response to the operation of creating a task in the second interactive interface, generating a task configuration file includes:

[0016] In response to an operation of configuring task information in the second interactive interface, acquiring task information;

[0017] In response to an operation of selecting a source data source in the second interactive interface, acquiring the source data source;

[0018] In response to an operation of selecting data to be synchronized in the second interaction interface, acquiring the data to be synchronized;

[0019] In response to an operation of selecting a synchronization rule in the second interactive interface, acquiring the synchronization rule;

[0020] In response to an operation of selecting a target data source in the second interactive interface, acquiring the target data source;

[0021] In response to the operation of configuring the execution cycle and frequency in the second interactive interface, acquiring the execution cycle and frequency;

[0022] In response to the operation of creating a data synchronization task in the second interactive interface, a target task is created, wherein the target task is used to synchronize the to-be-synchronized data in the source data source to the target data source according to the execution cycle and frequency and the synchronization rule.

[0023] Optionally, in response to the operation of starting the data synchronization task in the third interactive interface, synchronizing the to-be-synchronized data in the source data source to the target data source includes:

[0024] In response to an operation of starting a data synchronization task, determining a task type of the started task, and selecting a corresponding data synchronization plug-in according to the task type;

[0025] Parse the task configuration file to obtain the source data source, the target data source, the synchronization rule, the execution cycle and frequency, and the data to be synchronized;

[0026] Read the data to be synchronized from the source data source and determine whether to convert the data to be synchronized according to the synchronization rule;

[0027] In response to determining to convert the data to be synchronized, converting the data to be synchronized; in response to determining not to convert the data to be synchronized, outputting the converted data to the target data source;

[0028] Get the execution result data.

[0029] Optionally, the synchronization rule adopts hot update.

[0030] Optionally, the synchronization rule adopts hot update, including:

[0031] Test new synchronization rules;

[0032] When the new synchronization rule is correct, the new synchronization rule is processed into a target data packet in a target document format; the target document format matches the rule loading interface;

[0033] The target data packet is stored in a preset designated directory, so that a preset rule loading interface reads the target data packet and uses the synchronization rule corresponding to the target data packet.

[0034] According to a second aspect of an embodiment of the present disclosure, there is provided a data synchronization system, including a data source management module, a task management module and a background startup task module;

[0035] The data source management module is used to connect multiple data sources in response to the operation of configuring the data source in the first interactive interface;

[0036] The task management module is used to generate a task configuration file in response to an operation of creating a task in the second interactive interface;

[0037] The background task starting module is used to synchronize the to-be-synchronized data in the source data source to the target data source in response to the operation of starting the data synchronization task in the third interactive interface.

[0038] Optionally, the data source management module connects multiple data sources in response to an operation of configuring a data source in the first interactive interface, including:

[0039] In response to an operation of selecting a data source in the first interactive interface, obtaining a data source corresponding to the operation; the data source includes a source data source and a target data source;

[0040] In response to an operation of modifying a configuration parameter in the first interactive interface, acquiring a target configuration parameter;

[0041] In response to an operation of testing the connection between the source data source and the target data source in the first interactive interface, attempting to connect the source data source and the target data source;

[0042] In response to a successful connection, the data source is saved; or, in response to an unsuccessful connection, the step of obtaining the target configuration parameters is re-executed.

[0043] Optionally, the task management module updates the synchronization rule in a hot update manner, including:

[0044] In response to an operation indicating modification of a target configuration parameter, the target configuration parameter of a configuration table in a task configuration file is modified, and the task configuration file is stored in a target database.

[0045] Optionally, the task management module generates a task configuration file in response to an operation of creating a task in the second interactive interface, including:

[0046] In response to an operation of configuring task information in the second interactive interface, acquiring task information;

[0047] In response to an operation of selecting a source data source in the second interactive interface, acquiring the source data source;

[0048] In response to an operation of selecting data to be synchronized in the second interaction interface, acquiring the data to be synchronized;

[0049] In response to an operation of selecting a synchronization rule in the second interactive interface, acquiring the synchronization rule;

[0050] In response to an operation of selecting a target data source in the second interactive interface, acquiring the target data source;

[0051] In response to the operation of configuring the execution cycle and frequency in the second interactive interface, acquiring the execution cycle and frequency;

[0052] In response to the operation of creating a data synchronization task in the second interactive interface, a target task is created, wherein the target task is used to synchronize the to-be-synchronized data in the source data source to the target data source according to the execution cycle and frequency and the synchronization rule.

[0053] Optionally, the background task starting module synchronizes the to-be-synchronized data in the source data source to the target data source in response to the operation of starting the data synchronization task in the third interactive interface, including:

[0054] In response to an operation of starting a data synchronization task, determining a task type of the started task, and selecting a corresponding data synchronization plug-in according to the task type;

[0055] Parsing the task configuration file to obtain a source data source, a target data source, synchronization rules, an execution cycle and frequency, and data to be synchronized;

[0056] Reading the data to be synchronized from the source data source, and determining whether to convert the data to be synchronized according to the synchronization rule;

[0057] In response to determining to convert the data to be synchronized, converting the data to be synchronized; in response to determining not to convert the data to be synchronized, outputting the converted data to the target data source;

[0058] Get the execution result data.

[0059] Optionally, the task management module is further configured to update the synchronization rules in a hot update manner.

[0060] Optionally, the task management module updates the synchronization rule in a hot update manner, including:

[0061] Test new synchronization rules;

[0062] When the new synchronization rule is correct, the new synchronization rule is processed into a target data packet in a target document format; the target document format matches the rule loading interface;

[0063] The target data packet is stored in a preset designated directory, so that a preset rule loading interface reads the target data packet and uses the synchronization rule corresponding to the target data packet.

[0064] According to a third aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, and when an executable computer program in the storage medium is executed by a processor, the above-mentioned method can be implemented.

[0065] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:

[0066] As can be seen from the above embodiments, the solution provided by the embodiment of the present disclosure can connect multiple data sources in response to the operation of configuring the data source in the first interactive interface; then, in response to the operation of creating a task in the second interactive interface, a task configuration file is generated; thereafter, in response to the operation of starting the data synchronization task in the third interactive interface, the data to be synchronized in the source data source is synchronized to the target data source. In this way, in this embodiment, data transmission between different data sources can be realized through visual configuration operations without repeated development; and the configuration process is simple, which greatly reduces the learning cost.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0069] Figure 1The present invention is a flowchart of a data synchronization method according to an exemplary embodiment.

[0070] Figure 2 The figure is a flowchart of connecting a data source according to an exemplary embodiment.

[0071] Figure 3 The figure is a flowchart of generating a task configuration file according to an exemplary embodiment.

[0072] Figure 4 The figure is a flowchart of synchronizing data according to an exemplary embodiment.

[0073] Figure 5 The figure is a flowchart showing a hot update synchronization rule according to an exemplary embodiment.

[0074] Figure 6 The diagram is a schematic diagram of the architecture of a data synchronization system according to an exemplary embodiment.

[0075] Figure 7 The figure is a flowchart of obtaining a data source according to an exemplary embodiment.

[0076] Figure 8 The figure is a flowchart of creating a task according to an exemplary embodiment.

[0077] Fig. 9 The figure is a flowchart of synchronizing data according to an exemplary embodiment.

[0078] Fig.10 is a block diagram showing a time conversion rule according to an exemplary embodiment.

[0079] Fig.11 The figure is a block diagram showing a splitting and merging rule for conversion between fields according to an exemplary embodiment.

[0080] Fig.12 It is a block diagram showing a field mapping relationship rule according to an exemplary embodiment.

[0081] Fig.13 It is a block diagram of data desensitization rules according to an exemplary embodiment.

[0082] Fig.14 The figure is a flowchart of hot updating of synchronization rules according to an exemplary embodiment.

[0083] Fig.15 The figure is a flow chart showing a thermal update parameter according to an exemplary embodiment.

[0084] Fig.16 It is a block diagram of a data synchronization system according to an exemplary embodiment.

[0085] Fig.17 is a block diagram of another data synchronization system according to an exemplary embodiment.

[0086] Fig.18 is a block diagram of yet another data synchronization system according to an exemplary embodiment. DETAILED DESCRIPTION

[0087] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described exemplarily below do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices consistent with some aspects of the present disclosure as detailed in the attached claims. It should be noted that the features in the following embodiments and implementations may be combined with each other without conflict.

[0088] In order to solve the above technical problems, the present disclosure provides a data synchronization method, which can be applied to a data synchronization system. Figure 1 is a flow chart of a data synchronization method according to an exemplary embodiment. Figure 1 , a data synchronization method includes steps 11 to 13:

[0089] In step 11, in response to an operation of configuring a data source in the first interactive interface, a plurality of data sources are connected.

[0090] In this embodiment, an ETL component is pre-deployed in the data synchronization system, and the ETL component can read data from one data source and synchronize it to another data source to achieve the effect of synchronizing data. In one example, after the ETL component is started, an interactive interface can be displayed, which is subsequently referred to as the first interactive interface.

[0091] It should be noted that the first interactive interface, the second interactive interface and the third interactive interface can be three independent interactive interfaces, each of which corresponds to a corresponding function. For example, the first interactive interface only has the selection and candidate functions, and the second interactive interface only has the function of creating a task. The switching of the two interactive interfaces is achieved by pressing the previous or next button. In this way, the functions of the interactive interface are relatively simple, easier to operate and learn, and the difficulty of operation is reduced. Of course, the above-mentioned multiple interactive interfaces can also be integrated into one interface, and users can select the functions therein to use. In this way, the content of the interactive interface is more comprehensive, which is convenient for users to grasp the data synchronization method from a global perspective. Technicians can choose whether the first interactive interface, the second interactive interface and the third interactive interface are set separately or integrated into one interactive interface according to the specific scenario, and the corresponding scheme falls within the protection scope of this disclosure.

[0092] See also Figure 2 In step 21, the data synchronization system can respond to the operation of selecting a data source in the first interactive interface and obtain the data source corresponding to the operation. The first interactive interface includes options for data sources, and the above-mentioned data sources include source data sources and target data sources, wherein the source data source refers to the data source that provides the data to be synchronized, and can also be understood as the input source of the data synchronization system; the target data source refers to the data source that stores the data to be synchronized, and can also be understood as the output source of the data synchronization system. After selecting the source data source and the target data source, the data synchronization system can obtain the address (such as a URL address) or interface of the data source. It is understandable that after obtaining the address (such as a URL address) or interface of the data source, the data synchronization system can directly connect to the data source.

[0093] In this step, the data sources in the offline data synchronization scenario are shown in Table 1.

[0094] Table 1 Data sources used as input or output sources in offline data synchronization scenarios

[0095]

[0096]

[0097] In this step, the data sources in the real-time subscription scenario are shown in Table 2.

[0098] Table 2 Data sources used as input or output sources in real-time subscription scenarios

[0099] Data Source enter Output Mysql √ PostgresSql √ Clickhouse √ Mongodb √ Kafka √ √

[0100] In this step, the data source is divided into input source or output source, so that the user can choose the appropriate source data source and target data source during the configuration process. For example, in an offline scenario, you can choose Mysql, PostgresSql, Clickhouse, Mongodb, Hdfs, ElasticSearch and API as the source data source, and in a real-time order scenario, select Kafka as the source data source. Combining the scenario and the specific characteristics of each data source to divide it into input source and output source can ensure that the user selects a reliable data source and improve configuration efficiency.

[0101] In step 22, the data synchronization system can obtain the target configuration parameters in response to the operation of modifying the configuration parameters in the first interactive interface. The above-mentioned target configuration parameters include but are not limited to URL addresses, ports, etc., which can uniquely determine the parameters of the source data source or the target data source. In one example, the first interactive interface includes options for configuration parameters, and the user can trigger the above options to modify the configuration parameters. The data synchronization system can display a parameter menu in response to the operation of modifying the configuration parameters in the first interactive interface; when it is detected that the user selects a parameter in the parameter menu and a value of the parameter, the configuration parameter is obtained, which is subsequently referred to as the target configuration parameter. For example, when the user clicks on parameter A, a drop-down menu of parameter A can be displayed, and the candidate values ​​of parameter A will be displayed in the drop-down menu; the user can select a candidate value, so that the data synchronization system can save the candidate value selected by the user as the value of parameter A. For example, when the user clicks on parameter A, a sliding bar of parameter A can be displayed, with the two ends of the sliding bar representing the maximum and minimum values ​​of parameter A respectively; the user can slide the slider in the sliding bar, and the value corresponding to the current position can be displayed around the slider (such as above) during the sliding process; when the user clicks the OK button, the data synchronization system can save the value corresponding to the current position of the slider as the value of parameter A.

[0102] In step 23, the data synchronization system may try to connect the source data source and the target data source in response to the operation of testing the connection between the source data source and the target data source in the first interactive interface. In one example, the first interactive interface also includes an option for testing the connection. The user may operate the option for testing the connection to test whether data can be transmitted between the source data source and the target data source. When the data synchronization system detects the operation of testing the connection between the source data source and the target data source, it may connect to the source data source and the target data source respectively based on the address or interface of the data source in step 21, such as sending a test request to the source data source or the target data source, and receiving the response request returned by the source data source. If no response information is received within a set time (such as 20ms), it means that there is no connection. If a response information is received, it means that the connection is successful, so that it can be tested whether data can be transmitted between the source data source and the target data source. When data can be transmitted between the source data and the target data source, the data synchronization system may determine that the source data source and the target data source are connected. If the target data source of the source data source is not connected, the combination of the source data source and the target data source may be adjusted, or the addresses of the source data source and the target data source may be adjusted, so as to finally realize the connection between the source data source and the target data source and form a data synchronization channel.

[0103] In step 24, the data synchronization system may save the data source in response to successful connection. It is understandable that the data synchronization system may save source data sources, target data sources, target configuration parameters, connectable source data sources and target data source combinations, and the like.

[0104] It can be seen that in this step, the input source and output source of the data to be synchronized can be pre-configured into the data synchronization system, and the data synchronization channel can be established by selecting the corresponding input source and output source in the interactive interface before data synchronization. Compared with the development of corresponding data synchronization components in related technologies, only the above-mentioned data synchronization system needs to be installed, which improves the efficiency of data synchronization.

[0105] In step 12, in response to the operation of creating a task in the second interactive interface, a task configuration file is generated.

[0106] In this embodiment, after the source data source and the target data source are successfully connected and the data of the data source is saved, the data synchronization system can display an interactive interface containing options for creating a task, which is hereinafter referred to as the second interactive interface. Among them, the options for creating a task may include but are not limited to options for configuring task information, options for selecting a source data source, options for selecting data to be synchronized, options for selecting synchronization rules, options for selecting a target data source, options for configuring an execution cycle and frequency, and options for creating a data synchronization task. The options in the interactive interface can be selected according to the specific scenario.

[0107] In one example, after the second interactive interface displays the above options, the user can trigger the above options. The data synchronization system can detect operations in the second interactive interface, such as detecting operations to configure task information, select source data sources, select data to be synchronized, select synchronization rules, select target data sources, configure execution cycles and frequencies, or create data synchronization tasks.

[0108] See also Figure 3 In step 31, when an operation of configuring task information is detected, the data synchronization system can obtain task information in response to the operation of configuring task information in the second interactive interface. The task information may include but is not limited to the project, the business, etc., such as the data synchronization task of business B in project A.

[0109] In step 32, after acquiring the task information, the data synchronization system can continue to detect operations in the second interactive interface. When an operation of selecting a source data source is detected, the selected source data source can be acquired in response to the operation of selecting a source data source in the second interactive interface. It is understandable that in this step, one (or more) is selected from the data source list as the source data source. The above-mentioned data source list is created in step 11, that is, selecting a data source in step 11 is to select multiple data sources from several data sources as source data sources or target data sources to create a data source list.

[0110] In step 33, after acquiring the source data source, the data synchronization system may continue to detect operations within the second interactive interface. When an operation of selecting data to be synchronized is detected, the data synchronization system may respond to the operation of selecting data to be synchronized within the second interactive interface and acquire the selected data to be synchronized. That is, the data to be synchronized is the data in the selected source data source, such as the data to be synchronized may include but is not limited to databases, data tables, and data fields.

[0111] In step 34, after acquiring the data to be synchronized, the data synchronization system may continue to detect the operation in the second interactive interface, and when the operation of selecting a synchronization rule is detected, the selected synchronization rule may be acquired in response to the operation of selecting a synchronization rule in the second interactive interface. The data synchronization system may pre-set multiple synchronization rules, or the user may define a synchronization rule. For example, an SQL database may use a specified SQL statement and synchronize the results after execution.

[0112] In one example, the synchronization rule may include a time conversion rule. Fig.10 , the data synchronization system can convert the time in the data to be synchronized according to the above time conversion rules, such as time type conversion to obtain the current date / time, obtain the value of the time field year-month-day hour-minute-second, time addition and subtraction, and calculate the week, etc.; another example is time type conversion, converting the time in the data to be synchronized into a 10 / 13-digit timestamp, date, time, and time field, etc.; another example is time format conversion, converting the time format into yyyy-MM-dd HH:mm:ss (such as 2022-03-01 13:15:18), yyyyMMdd (such as 20220301), yyyy / MM / dd (such as 2022 / 03 / 01), etc. In this example, the time of the data to be synchronized can be converted into time data of different types or formats to meet the needs of different businesses.

[0113] In another example, the synchronization rule may include a field conversion splitting and merging rule. Fig.11, the above-mentioned inter-field conversion split-merge rules may include field merging, such as through certain character splicing, field (value) calculation, and merging in json format. Taking character splicing as an example, for example, the data columns "surname" and "name" can be merged into one column as "name", and the data columns "grade" and "class" can be merged into one column as "grade / class". For another example, the above-mentioned inter-field conversion split-merge rules may include field splitting, such as splitting by position, splitting according to a certain paragraph of text, and field interception. Taking splitting by position as an example, the data column "grade / class" is split into 2 columns "grade" and "class". For another example, the above-mentioned inter-field conversion split-merge rules may include field conversion, such as filtering, aggregation, string conversion, etc. Taking string conversion, for example, converting uppercase letters into lowercase letters, converting pinyin into Chinese characters, etc. In this example, the field conversion, splitting, merging, etc. in the data to be synchronized can be processed to obtain the required types of data to meet the needs of various businesses and improve the utilization of data.

[0114] In another embodiment, the synchronization rules may include field mapping relationship rules. Fig.12 , the above-mentioned field mapping relationship rules may include the mapping of the output field content and the original field content, for example, the numerical value in the original field content is respectively mapped with the numerical value, data / time interval and character string in the output field content; the data / time interval in the original field content is respectively mapped with the numerical value, data / time interval and character string in the output field content; the character string in the original field content is respectively mapped with the numerical value, data / time interval and character string in the output field content. For example, the age data age is converted into the age stage stage, such as the age of 1-10 years old corresponds to the age stage infants, the age of 11-18 years old corresponds to the age stage teenagers, etc. This embodiment can obtain different field contents through the field mapping relationship rules, so as to meet the needs of various businesses and improve the utilization rate of data.

[0115] In another embodiment, the synchronization rule may further include a data masking type. Fig.13 , field desensitization rules may include basic desensitization types, sensitive field types, and desensitization types. Among them, basic desensitization types include but are not limited to ID number, password, mobile phone number / email, etc.; sensitive field types include but are not limited to sensitive content, privacy information, etc.; desensitization types may include but are not limited to selecting basic desensitization types, custom desensitization rules, and automatic checking of sensitive fields, etc. For example, the mobile phone number is desensitized to 139****0588. In this embodiment, the data desensitization rules can ensure the security of the data synchronization process and avoid information leakage.

[0116] It should be noted that in this step, the synchronization rule may include a first preset rule, which refers to the rule for processing the data to be synchronized itself, which is used for each piece of data to be synchronized, such as time conversion, field splitting and merging, field mapping, sensitive field desensitization, etc.; it may also include a second preset rule, which refers to the rule for processing a preset number (e.g., hundreds to tens of thousands) of data to be synchronized using a preset model (e.g., a neural network model, a speech processing model, a text processing model, etc.), which is used for batch processing, such as extracting events or semantic parsing in the data to be synchronized. The technician can select the appropriate synchronization rule according to the specific scenario, and the corresponding scheme falls within the protection scope of the present disclosure.

[0117] In step 35, after obtaining the synchronization rules, the data synchronization system can continue to detect the operation in the second interactive interface, and when the operation of selecting the target data source is detected, the selected target data source can be obtained in response to the operation of selecting the target data source in the second interactive interface. It is understandable that the above target data source can be selected from the above data source list, and the specific content can refer to the method of obtaining the source data source.

[0118] In step 36, after obtaining the synchronization rules, the data synchronization system can continue to detect operations in the second interactive interface. When an operation to configure the execution cycle and frequency is detected, the configured execution cycle and frequency can be obtained in response to the operation to configure the execution cycle and frequency in the second interactive interface. The above-mentioned execution cycle may include but is not limited to one day, one week or one month, and the above-mentioned frequency may include but is not limited to once a day (or multiple times), once a week (or multiple times) or once a month (or multiple times). It is understandable that the above-mentioned execution cycle and frequency can be set selectively, for example, if the execution cycle is set, there is no need to set the frequency, or if the frequency is set, there is no need to set the cycle, etc., which can be set according to the specific scenario. In this way, in this step, by configuring the execution cycle and frequency, the data transmission between the source database and the target data can be matched to ensure transmission efficiency; and it can also ensure that data synchronization can meet the data transmission requirements of the data synchronization task.

[0119] In step 37, after obtaining the execution cycle and frequency, the data synchronization system can continue to detect operations in the second interactive interface. When an operation to create a data synchronization task is detected, the data synchronization task, i.e., the target task, can be created in response to the operation to create a data synchronization task in the second interactive interface. It is understandable that the above-mentioned target task is used to synchronize the data to be synchronized in the source data source to the target data source according to the above-mentioned execution cycle and frequency and synchronization rules. In this way, the data synchronization system can visually display various options in the second interactive interface, and the user can create the target task through visual operations, which is simple, fast and efficient.

[0120] In step 13, in response to the operation of starting the data synchronization task in the third interactive interface, the data to be synchronized in the source data source is synchronized to the target data source.

[0121] In this embodiment, after creating a data synchronization task, i.e., a target task, the data synchronization system may display an interactive interface including an option to start the data synchronization task, which is hereinafter referred to as the third interactive interface. The user may select a task list of the data synchronization task, select at least one task from the task list, and choose to start the data synchronization task. The data synchronization system may detect an operation in the third interactive interface, and when an operation to start the data synchronization task is detected, the data to be synchronized in the source data source may be synchronized to the target data source in response to the operation to start the data synchronization task in the third interactive interface.

[0122] See also Figure 4 In step 41, the data synchronization system can respond to the operation of starting the data synchronization task, determine the task type of the started task, and select the corresponding data synchronization plug-in according to the task type. The data synchronization plug-in is the data ETL conversion tool mentioned in the above embodiment.

[0123] In step 42, the data synchronization system may parse the task configuration file to obtain the source data source, target data source, synchronization rules, execution cycle and frequency, and data to be synchronized in step 11.

[0124] In step 43, the data synchronization plug-in may read the data to be synchronized from the source data source, and determine whether to convert the data to be synchronized according to a preset synchronization rule.

[0125] In step 44, when it is determined that the data to be synchronized needs to be converted, the data synchronization plug-in can convert the data to be synchronized in response to the determination to convert the data to be synchronized; in response to the determination not to convert the data to be synchronized, the converted data is output to the target data source. In step 45, the data synchronization system can obtain the execution result data and write the execution result data to the synchronization record.

[0126] In one embodiment, taking into account the actual need of users to update synchronization rules during the data synchronization process, the synchronization rules in the data synchronization system in this embodiment adopt a hot update method, that is, the synchronization rules are updated without affecting data synchronization and after the synchronization rules are updated, they can replace the synchronization rules before the update and put into use.

[0127] In one example, see Figure 5 In step 51, the data synchronization system may obtain new synchronization rules and test the new synchronization rules.

[0128] In step 52, the data synchronization system may process the new synchronization rule into a target data packet in a target document format (such as jar format) when the new synchronization rule is correct, and the target document format matches the rule loading interface.

[0129] In step 53, the data synchronization system can store the target data packet in a preset designated directory (such as the transformPlugin target) so that the preset rule loading interface reads the target data packet and uses the synchronization rules corresponding to the target data packet. For example, in this example, the new synchronization rule can be processed into a jar package, and the jar package can be placed under the transformPlugin target. During the data synchronization process, the rule loading interface can load all jar packages under the above transformPlugin target. Then, the data synchronization system can determine whether the synchronization rule can be used normally. If it cannot be used normally, roll back and delete the newly synchronized jar package, and reload the rule jar package. If it can be used normally, the hot update ends. In this step, the data synchronization system can generate a task progress bar (or task completion ratio) during the data synchronization process, and the ratio or sliding in the task progress bar indicates the proportion of the currently synchronized data to the total data; when the task is completed, the data synchronization system can generate a task completion mark, such as a task progress bar showing 100%, completed, or switching to a completed menu, so that users can understand the task progress in a timely manner. In this way, in this example, the synchronization rules are updated by hot update, and the rules can be adjusted without affecting the data synchronization process, which is conducive to improving data synchronization efficiency and improving user experience.

[0130] In another example, the data synchronization system can modify the target configuration parameters of the configuration table in the task configuration file in response to the operation of the target configuration parameters, and the above task configuration file is stored in the target database, which can be a local database corresponding to the data synchronization system. In this way, the hot update method is used to update the target configuration parameters in this example, which can ensure that the configuration parameters are effective without stopping the data synchronization task, which is conducive to improving data synchronization efficiency and improving user experience.

[0131] So far, the solution provided by the embodiment of the present disclosure can connect multiple data sources in response to the operation of configuring the data source in the first interactive interface; then, in response to the operation of creating a task in the second interactive interface, a task configuration file is generated; thereafter, in response to the operation of starting the data synchronization task in the third interactive interface, the data to be synchronized in the source data source is synchronized to the target data source. In this way, in this embodiment, data transmission between different data sources can be realized through visual configuration operations without repeated development; and the configuration process is simple, which greatly reduces the learning cost.

[0132] The working principle of the data synchronization system is described below in conjunction with an embodiment. Figure 6 ,include:

[0133] 1. Data source creation process

[0134] See also Figure 7 , the data synchronization system can display a first interactive interface. In the first interactive interface, the user can select a data source, including a source data source and a target data source, such as Mysql, Hdfs, and API, etc. The data synchronization system can display prompt information in the first interactive interface, so that the user can fill in the corresponding configuration parameters according to the prompt information in the first interactive interface to obtain the target configuration parameters. Then, the data synchronization system can test the connection. If the connection is successful, the data synchronization system can save the data source; if the connection fails, the user is prompted to check the configuration parameters until the connection is successful before the data source can be saved.

[0135] 2. Task creation process

[0136] See also Figure 8 , the data synchronization system can display a second interactive interface and generate task information that prompts the user to perform the task creation process, such as the project, business, etc. Then, select the data source for data input, i.e., the source data source, where the source data source is selected from the data source that has been created in the above step (1). Therefore, if there is no required data source, it is necessary to return to step (1) to create a data source.

[0137] Then, after selecting the source data source, you can view the data stored in the data source, select the data to be synchronized according to your needs (such as selecting databases, tables, and fields, etc.), and further formulate data synchronization rules based on the data source type (for example, a SQL database can specify SQL statements and synchronize the results after execution).

[0138] After that, select the output data source, i.e. the target data source, and configure the execution cycle and frequency to create the task.

[0139] 3. Task execution process

[0140] See also Fig. 9 After the task is created, the data synchronization system will generate a task configuration file. The data synchronization system can display the third interactive interface and start the task according to the user's operation to start the task. At this time, the data synchronization system can choose to start the offline synchronization plug-in or the real-time subscription plug-in according to the task type. After the task is started, the configuration file is parsed, the configuration parameters of the connected data source are obtained, and the data to be synchronized is read in. After the data to be synchronized is read in, it is determined whether data conversion is required based on the configuration file. If not, the data to be synchronized is directly output to the target data source. If necessary, it is output after the data conversion process; after the execution is completed, the execution result is returned and displayed on the third interactive interface.

[0141] 4. Task synchronization rules

[0142] The synchronization rules may include a first preset rule and a second preset rule, wherein the first preset rule is used to process the data to be synchronized, such as time conversion, field splitting and merging, field mapping relationship, sensitive field desensitization, etc. The second preset rule is used to process a preset number (e.g., hundreds to tens of thousands) of data to be synchronized using a preset model (e.g., a neural network model, a speech processing model, a text processing model, etc.), that is, a rule containing a preset model, which is used for batch processing, such as extracting events or semantic analysis from the data to be synchronized. The technician can select appropriate synchronization rules according to the specific scenario, and the corresponding scheme falls within the protection scope of the present disclosure.

[0143] (1) Time conversion rules

[0144] See also Fig.10 , time conversion rules may include time format conversion, time type conversion and time type conversion, etc. Among them, time format conversion may include but is not limited to yyyy-MM-dd HH:mm:ss format, yyyyMMdd format and yyyy / MM / dd format, etc.; time type conversion may include but is not limited to 10 / 13-bit timestamp, date, time, time field, etc.; time type conversion may include but is not limited to obtaining the current date / time, obtaining the value of the time field year-month-day hour-minute-second, time addition and subtraction, and calculation of the week, etc. The technicians can make a selection according to the specific scenario, and the corresponding scheme falls within the protection scope of the present disclosure.

[0145] In one example, the 13-bit timestamp field event_time is converted to event_date in the format of yyyy-MM-dd HH:mm:ss, as shown in Table 3.

[0146] Table 3 Timestamp and specific time conversion

[0147] event_time event_date 1627351547610 2021-07-27 10:05:47

[0148] (2) Rules for splitting and merging fields

[0149] See also Fig.11 The field conversion splitting and merging rules include field conversion rules, field splitting rules and field merging rules. The field conversion rules include but are not limited to filtering fields, aggregating fields, string conversion, etc. The field splitting rules include but are not limited to splitting by position, splitting by a certain character segment and field interception, etc. The field merging rules include but are not limited to splicing through certain characters, calculating between fields (values), merging in json format, etc.

[0150] In one example, the string str is split into str1 and str2 fields according to “-”, as shown in Table 4.

[0151] Table 4 String splitting rules

[0152]

[0153]

[0154] (3) Field mapping rules

[0155] See also Fig.12 The field mapping rule is the mapping of the original field and the output field content. For example, the numerical value in the original field content is mapped to the numerical value, data / time interval and string in the output field content; the data / time interval in the original field content is mapped to the numerical value, data / time interval and string in the output field content; the string in the original field content is mapped to the numerical value, data / time interval and string in the output field content.

[0156] In one example, the age data age is converted into age stage stage, as shown in Table 5.

[0157] (4) Field desensitization rules

[0158] See also Fig.13 , field desensitization rules can include basic desensitization types, sensitive field types, and desensitization types. Among them, basic desensitization types include but are not limited to ID number, password, mobile phone number / email address, etc.; sensitive field types include but are not limited to sensitive content, privacy information, etc.; desensitization types can include but are not limited to selecting basic desensitization types, custom desensitization rules, and automatic checking of sensitive fields, etc. For example, desensitize the mobile phone number to 139****0588.

[0159] Table 5 Conversion rules between age data and age stage

[0160] age stage 1-10 Infants and young children 11-18 teenager 19-35 youth 36-50 Middle Age 51-65 middle aged >65 elderly

[0161] (5) Synchronous rule hot update

[0162] See also Fig.14 In order to prevent the synchronization rule update process from affecting the data synchronization business, the synchronization rule can be packaged as a detachable rule component. After completing the development of a synchronization rule, you can put the jar package of the synchronization rule in the transformPlugin directory and request to load the rule loading interface of the synchronization rule to load all the jar packages in the directory. Then observe whether the synchronization rules on the data synchronization system are normal. If not, roll back, delete the jar package, and request to reload the rule jar package; if normal, the hot update is completed.

[0163] (6) Hot update of task configuration parameters

[0164] See also Fig.15 In this example, for real-time subscription tasks, unlike offline tasks, real-time tasks will not stop once they start. When the user wants to modify the task configuration file, the data synchronization rules are stored in the target database. When modifying the synchronization rules, the configuration table is directly updated. During the execution of the data synchronization task, in order to take effect without stopping the task, the synchronization rules in the configuration table are read regularly in the program. If the configuration table has not changed, it does not need to be changed and it will be executed normally. If the configuration table has changed, it can be adjusted immediately and then continue to execute.

[0165] The working principle of the data synchronization system is described below in conjunction with another embodiment. Fig.16 and Fig.17 ,include:

[0166] 0. Preparation

[0167] Develop data source management (add, modify, delete, etc.), task management (add, modify, delete, publish, log view, etc.), permission management, user management functions and service interfaces. The above functions can be completed in the interactive page; develop image creation and image publishing functions based on docker api; develop task deployment and task log view functions based on k8s api; develop task management server and task execution client functions based on grpc protocol. The server distributes tasks to the execution client. The client can complete task execution in the local environment and k8s environment according to different configuration strategies, and can also perform customized development to complete task execution based on the environment. The above WEB end and back end (i.e., server and client) constitute the data synchronization system in the embodiment of the present disclosure.

[0168] 1. Configure the data source

[0169] Add data source, support mysql / postgresql / clickhouse / hive / elasticsearch / restapi / mongodb / redis / hdfs / kafka data source link test and addition. Enter the data source ip, address, port, username, password and other information. After authentication, save the data source information to the mysql database. Each user who logs in to the system can only see and use the data source they created.

[0170] 2. Select the source data source and the target data source to complete the data synchronization task creation

[0171] Select the source data source and data table (the data table supports all fields or some fields) from the list of configured data sources in (1), select the target data source and data table from the list of configured data sources, and complete the configuration of the source data source and the target data source. The data synchronization system completes the flinkx-based configuration file according to the source data source and the target data source configuration for subsequent execution of flinkx tasks. It is understandable that the above flinkx is only one of the methods, and data synchronization can also be completed by using, for example, the DataX method, and the corresponding solution also falls within the protection scope of this disclosure.

[0172] For example, fill in the task name, task scheduling time (supports both single execution and daily scheduled execution), complete task creation, and save the data in the MySQL database.

[0173] The task scheduling module starts the grpc server and waits for 1-N clients to connect to the task execution. The task scheduling is responsible for scanning the unexecuted tasks in the database, and the scan is performed every 10 seconds. After discovering a new task, a client is selected from 1-N clients (randomly or by polling algorithm), and the server-client task distribution is completed through the grpc protocol. The task execution client may exit abnormally. At this time, the task execution server needs to release the client to ensure that subsequent new tasks are not distributed to the client.

[0174] Execute running tasks, call k8s API to query task execution status (running, completed, error), modify the task status and task log in the database for subsequent log query and task monitoring.

[0175] 3. Image building

[0176] After receiving the new task, the task execution client generates a task configuration file based on flinkx, builds the task image through the task configuration file + the image based on flinkx, and publishes the built image to the harbor image repository.

[0177] 4. Task deployment

[0178] After the server is started, the client is started. After the client is started, it establishes a grpc link with the server to complete the client registration. After the server notifies that there is a new task, the client generates a task configuration file based on flinkx, generates a new image based on the flinkx image, and uploads it to the harbor image repository. Send a control instruction to the k8s server, which includes the selected deployment mode and the address of the Flinkx image in Harbor, that is, call the k8s api to complete the task image deployment. K8s can download the Flinkx image from the Harbor warehouse according to the address and deploy it. After the deployment is successful, it automatically executes the task according to the configuration file. During the task execution process and after the task is completed, the data collection module in k8s can collect various types of data, such as task execution time, end time, number of synchronized data, used traffic, etc., and send the above types of data to the Kafka message component. The task status monitoring module in the Gprc server can monitor the Kafka message component. When a new message (i.e. various types of data) is monitored, it can consume the above message and write it back to the Mysql database, that is, adjust various statuses of the task, such as task progress, number of completed data, start time, end time, etc., so as to facilitate users to view task information from the interactive interface of the WEB side.

[0179] 5. Task monitoring

[0180] The UI operation page of the data synchronization system, i.e. the interactive interface, can view the detailed information of each task, including the start execution time, end execution time, execution environment, execution status, execution log and other information.

[0181] The present disclosure also provides a data synchronization system. Fig.18 , including: a data source management module 181, a task management module 182 and a background startup task module 183;

[0182] The data source management module 181 is used to connect multiple data sources in response to the operation of configuring the data source in the first interactive interface;

[0183] The task management module 182 is used to generate a task configuration file in response to the operation of creating a task in the second interactive interface;

[0184] The background task starting module 183 is used to synchronize the data to be synchronized in the source data source to the target data source in response to the operation of starting the data synchronization task in the third interactive interface.

[0185] In one embodiment, the data source management module connects multiple data sources in response to an operation of configuring a data source in the first interactive interface, including:

[0186] In response to an operation of selecting a data source in the first interactive interface, obtaining a data source corresponding to the operation; the data source includes a source data source and a target data source;

[0187] In response to an operation of modifying a configuration parameter in the first interactive interface, acquiring a target configuration parameter;

[0188] In response to an operation of testing the connection between the source data source and the target data source in the first interactive interface, attempting to connect the source data source and the target data source;

[0189] In response to a successful connection, the data source is saved; or, in response to an unsuccessful connection, the step of obtaining the target configuration parameters is re-executed.

[0190] In one embodiment, the task management module updates the synchronization rules in a hot update manner, including:

[0191] In response to an operation indicating modification of a target configuration parameter, the target configuration parameter of a configuration table in a task configuration file is modified, and the task configuration file is stored in a target database.

[0192] In one embodiment, the task management module generates a task configuration file in response to an operation of creating a task in the second interactive interface, including:

[0193] In response to an operation of configuring task information in the second interactive interface, acquiring task information;

[0194] In response to an operation of selecting a source data source in the second interactive interface, acquiring the source data source;

[0195] In response to an operation of selecting data to be synchronized in the second interaction interface, acquiring the data to be synchronized;

[0196] In response to an operation of selecting a synchronization rule in the second interactive interface, acquiring the synchronization rule;

[0197] In response to an operation of selecting a target data source in the second interactive interface, acquiring the target data source;

[0198] In response to the operation of configuring the execution cycle and frequency in the second interactive interface, acquiring the execution cycle and frequency;

[0199] In response to the operation of creating a data synchronization task in the second interactive interface, a target task is created, wherein the target task is used to synchronize the to-be-synchronized data in the source data source to the target data source according to the execution cycle and frequency and the synchronization rule.

[0200] In one embodiment, the background task initiation module synchronizes the to-be-synchronized data in the source data source to the target data source in response to the operation of initiating the data synchronization task in the third interactive interface, including:

[0201] In response to an operation of starting a data synchronization task, determining a task type of the started task, and selecting a corresponding data synchronization plug-in according to the task type;

[0202] Parsing the task configuration file to obtain a source data source, a target data source, synchronization rules, an execution cycle and frequency, and data to be synchronized;

[0203] Reading the data to be synchronized from the source data source, and determining whether to convert the data to be synchronized according to the synchronization rule;

[0204] In response to determining to convert the data to be synchronized, converting the data to be synchronized; in response to determining not to convert the data to be synchronized, outputting the converted data to the target data source;

[0205] Get the execution result data.

[0206] In one embodiment, the task management module is further configured to update the synchronization rules in a hot update manner.

[0207] In one embodiment, the task management module updates the synchronization rules in a hot update manner, including:

[0208] Test new synchronization rules;

[0209] When the new synchronization rule is correct, the new synchronization rule is processed into a target data packet in a target document format; the target document format matches the rule loading interface;

[0210] The target data packet is stored in a preset designated directory, so that a preset rule loading interface reads the target data packet and uses the synchronization rule corresponding to the target data packet.

[0211] It should be noted that the system shown in this embodiment matches the contents of the method embodiment, and reference may be made to the contents of the above system embodiment, which will not be repeated here.

[0212] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including an executable computer program, and the executable computer program can be executed by a processor to implement the following. Figure 3 The method of the embodiment shown. The readable storage medium may be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0213] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0214] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data synchronization method, characterized in that: include: In response to an operation of configuring a data source in the first interactive interface, connecting a plurality of data sources; In response to an operation of creating a task in the second interactive interface, generating a task configuration file; In response to the operation of starting the data synchronization task in the third interactive interface, synchronizing the to-be-synchronized data in the source data source to the target data source; In response to the operation of creating a task in the second interactive interface, a task configuration file is generated, including: In response to an operation of configuring task information in the second interactive interface, acquiring task information; In response to an operation of selecting a source data source in the second interactive interface, acquiring the source data source; In response to an operation of selecting data to be synchronized in the second interaction interface, acquiring the data to be synchronized; In response to an operation of selecting a synchronization rule in the second interactive interface, acquiring the synchronization rule; In response to an operation of selecting a target data source in the second interactive interface, acquiring the target data source; In response to the operation of configuring the execution cycle and frequency in the second interactive interface, acquiring the execution cycle and frequency; In response to an operation of creating a data synchronization task in the second interactive interface, creating a target task, wherein the target task is used to synchronize the to-be-synchronized data in the source data source to the target data source according to the execution cycle and frequency and the synchronization rule; In response to the operation of starting the data synchronization task in the third interactive interface, synchronizing the to-be-synchronized data in the source data source to the target data source includes: In response to an operation of starting a data synchronization task, determining a task type of the started task, and selecting a corresponding data synchronization plug-in according to the task type; Parse the task configuration file to obtain the source data source, the target data source, the synchronization rule, the execution cycle and frequency, and the data to be synchronized; Read the data to be synchronized from the source data source and determine whether to convert the data to be synchronized according to the synchronization rule; In response to determining to convert the data to be synchronized, converting the data to be synchronized; in response to determining not to convert the data to be synchronized, outputting the converted data to the target data source; Get the execution result data.

2. The method according to claim 1, characterized in that In response to the operation of configuring a data source in the first interactive interface, connecting multiple data sources includes: In response to an operation of selecting a data source in the first interactive interface, obtaining a data source corresponding to the operation; the data source includes a source data source and a target data source; In response to an operation of modifying a configuration parameter in the first interactive interface, acquiring a target configuration parameter; In response to an operation of testing the connection between the source data source and the target data source in the first interactive interface, attempting to connect the source data source and the target data source; In response to successful connection, the data source is saved.

3. The method according to claim 2, characterized in that The synchronization rule adopts hot update, including: In response to an operation indicating modification of a target configuration parameter, the target configuration parameter of a configuration table in a task configuration file is modified, and the task configuration file is stored in a target database.

4. The method according to claim 1, characterized in that: The synchronization rule adopts hot update.

5. The method according to claim 4, characterized in that The synchronization rule adopts hot update, including: Test new synchronization rules; When the new synchronization rule is correct, the new synchronization rule is processed into a target data packet in a target document format; the target document format matches the rule loading interface; The target data packet is stored in a preset designated directory, so that a preset rule loading interface reads the target data packet and uses the synchronization rule corresponding to the target data packet.

6. A data synchronization system, characterized in that: include: Data source management module, task management module and background startup task module; The data source management module is used to connect multiple data sources in response to the operation of configuring the data source in the first interactive interface; The task management module is used to generate a task configuration file in response to an operation of creating a task in the second interactive interface; The background task starting module is used to synchronize the data to be synchronized in the source data source to the target data source in response to the operation of starting the data synchronization task in the third interactive interface; The task management module generates a task configuration file in response to an operation of creating a task in the second interactive interface, including: In response to an operation of configuring task information in the second interactive interface, acquiring task information; In response to an operation of selecting a source data source in the second interactive interface, acquiring the source data source; In response to an operation of selecting data to be synchronized in the second interaction interface, acquiring the data to be synchronized; In response to an operation of selecting a synchronization rule in the second interactive interface, acquiring the synchronization rule; In response to an operation of selecting a target data source in the second interactive interface, acquiring the target data source; In response to the operation of configuring the execution cycle and frequency in the second interactive interface, acquiring the execution cycle and frequency; In response to an operation of creating a data synchronization task in the second interactive interface, creating a target task, wherein the target task is used to synchronize the to-be-synchronized data in the source data source to the target data source according to the execution cycle and frequency and the synchronization rule; The background task starting module synchronizes the to-be-synchronized data in the source data source to the target data source in response to the operation of starting the data synchronization task in the third interactive interface, including: In response to an operation of starting a data synchronization task, determining a task type of the started task, and selecting a corresponding data synchronization plug-in according to the task type; Parsing the task configuration file to obtain a source data source, a target data source, synchronization rules, an execution cycle and frequency, and data to be synchronized; Reading the data to be synchronized from the source data source, and determining whether to convert the data to be synchronized according to the synchronization rule; In response to determining to convert the data to be synchronized, converting the data to be synchronized; in response to determining not to convert the data to be synchronized, outputting the converted data to the target data source; Get the execution result data.

7. The system according to claim 6, characterized in that The data source management module connects multiple data sources in response to the operation of configuring the data source in the first interactive interface, including: In response to an operation of selecting a data source in the first interactive interface, obtaining a data source corresponding to the operation; the data source includes a source data source and a target data source; In response to an operation of modifying a configuration parameter in the first interactive interface, acquiring a target configuration parameter; In response to an operation of testing the connection between the source data source and the target data source in the first interactive interface, attempting to connect the source data source and the target data source; In response to a successful connection, the data source is saved; or, in response to an unsuccessful connection, the step of obtaining the target configuration parameters is re-executed.

8. The system according to claim 7, characterized in that The task management module updates the synchronization rules in a hot update manner, including: In response to an operation indicating modification of a target configuration parameter, the target configuration parameter of a configuration table in a task configuration file is modified, and the task configuration file is stored in a target database.

9. The system according to claim 6, characterized in that The task management module is also used to update the synchronization rules in a hot update manner.

10. The system according to claim 9, characterized in that The task management module updates the synchronization rules in a hot update manner, including: Test new synchronization rules; When the new synchronization rule is correct, the new synchronization rule is processed into a target data packet in a target document format; the target document format matches the rule loading interface; The target data packet is stored in a preset designated directory, so that a preset rule loading interface reads the target data packet and uses the synchronization rule corresponding to the target data packet.

11. A non-transitory computer-readable storage medium, characterized in that: When the executable computer program in the storage medium is executed by a processor, the method according to any one of claims 1 to 5 can be implemented.

Images