Data heterogeneous synchronous management method and device, storage medium and electronic device

Abstract

The invention discloses a data heterogeneous synchronization management method and device, a storage medium and an electronic device, and relates to the technical field of smart home, and the data heterogeneous synchronization management method comprises the steps: obtaining a to-be-synchronized source database from a user maintenance data source list; selecting an original data field in the source database and a target data field in a target database, and configuring a mapping relation of the data fields; according to a preset data synchronization rule, constructing a synchronization task script according to the mapping relation; and creating a data synchronization job task according to the synchronization task script, and sending the data synchronization job task to a job scheduling engine, so that the job scheduling engine performs data synchronization operation. According to the technical scheme, the data synchronization efficiency and accuracy are improved, and the back-end interface development data access development communication cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of smart home, and in particular, to a method, device, storage medium and electronic device for heterogeneous data synchronization management. Background Art

[0002] With the digital transformation in all walks of life, data plays a dominant role in enterprise development. Data synchronization and migration have become important links in enterprise data processing. Traditional data synchronization often relies on complex manual script writing configuration or application interface docking, which is inefficient and error-prone. Designing and developing a visual and autonomous data synchronization method suitable for developers, testers, and even data operation personnel can more effectively assist business development.

[0003] In the scenario of the digital application system of the online mall, there are various scenarios, and multi-dimensional data such as people, products, stores, and domains are isolated from each other. There are different business data barriers, and the data scope involved in business requirements is not only the data of a single business, but cross-business and multi-source data. For similar multi-source data business requirements, on the one hand, develop backend interfaces to pull data, and indirectly achieve cross-business group mutual access to data. On the other hand, you can manually write and build corresponding plug-in data synchronization scripts and upload them for scheduling and execution. Both of them require development intervention for repetitive development operations. Based on the multi-instance library of internal business and the cross-business data information barrier, for multi-related query business requirements, the repetitive development operation has low efficiency and long cycle, and is not sufficient to cope with the rapid iteration demand development tasks. Summary of the Invention

[0004] Embodiments of the present application provide a method, device, storage medium and electronic device for heterogeneous data synchronization management, aiming to automatically construct a synchronization task script for data synchronization for different data sources and achieve efficient synchronization of multi-source data.

[0005] According to one aspect of the embodiments of the present application, a method for heterogeneous data synchronization management is provided, which can be applied to a user terminal device. The method for heterogeneous data synchronization management includes:

[0006] Obtain the source database to be synchronized from the data source list maintained by the user;

[0007] Select the original data fields in the source database and the target data fields in the target database, and configure the mapping relationship of the data fields;

[0008] Construct a synchronization task script according to the mapping relationship according to the preset data synchronization rules;

[0009] Create a data synchronization job task according to the synchronization task script and send it to the job scheduling engine so that the job scheduling engine performs data synchronization operations.

[0010] According to an implementation manner of an embodiment of the present application, before obtaining the source database to be synchronized from the user-maintained data source list, it further includes:

[0011] Configure the data source parameter information of the source database to generate a maintained data source item;

[0012] Add the maintained data source item to the user-maintained data source list under the user account.

[0013] According to an implementation manner of an embodiment of the present application, the configuring the data source parameter information of the source database to generate a maintained data source item includes:

[0014] Configure the data source type, data source name, user account, user password, database uniform resource locator URL, and database JDBC driver class of the source database.

[0015] According to an implementation manner of an embodiment of the present application, the selecting the original data fields in the source database and the target data fields in the target database and configuring the mapping relationship of the data fields includes:

[0016] Select the source-side database table in the source database as the original data field;

[0017] Select the target-side database table in the target database as the target data field;

[0018] Configure the mapping relationship of at least some fields from the source-side database table to the target-side database table.

[0019] According to an implementation manner of an embodiment of the present application, the selecting the source-side database table in the source database as the original data field includes:

[0020] Select the source database and the database table name serving as the source-side database table therein;

[0021] Write the pre-SQL statement and post-SQL statement of the data fields to be synchronized in the source-side database table to determine the original data field in the source-side database table.

[0022] According to an implementation manner of an embodiment of the present application, the constructing a synchronization task script according to the mapping relationship according to a preset data synchronization rule includes:

[0023] Determine the reading tool of the source database and the writing tool of the target-side database table;

[0024] Write the connection information of the reading tool and the writing tool, the source database table to be synchronized, its pre-SQL statement and post-SQL statement, and the target database table and the target data fields into a configuration file;

[0025] Use a Python script to read the configuration file and automatically generate a JSON file as the synchronization task script.

[0026] According to an implementable manner of the embodiment of the present application, after creating a data synchronization job task according to the synchronization task script and sending it to the job scheduling engine, it further includes:

[0027] Set a timing schedule for the data synchronization job task so that the data synchronization job task is executed according to the set time;

[0028] Obtain the job execution log, determine the execution status of the data synchronization job task and display it to the user.

[0029] According to another aspect of the embodiment of the present application, there is provided a data heterogeneous synchronization management device configured in a user terminal device. The data heterogeneous synchronization management device includes:

[0030] A source database acquisition module for acquiring a source database to be synchronized from a user-maintained data source list;

[0031] A mapping relationship configuration module for selecting original data fields in the source database and target data fields in the target database and configuring the mapping relationship of the data fields;

[0032] A task script construction module for constructing a synchronization task script according to the mapping relationship according to a preset data synchronization rule;

[0033] A synchronization job task sending module for creating a data synchronization job task according to the synchronization task script and sending it to the job scheduling engine for the job scheduling engine to perform data synchronization operations.

[0034] According to an implementable manner of the embodiment of the present application, the data heterogeneous synchronization management device further includes:

[0035] A maintenance data source project generation module for configuring data source parameter information of the source database and generating a maintenance data source project before acquiring the source database to be synchronized from the user-maintained data source list;

[0036] A user-maintained data source list determination module for adding the maintenance data source project to the user-maintained data source list under the user account.

[0037] According to an implementable manner of an embodiment of the present application, the maintenance data source item generation module is specifically configured to:

[0038] Configure the data source type, data source name, user account, user password, database uniform resource locator URL, and database JDBC driver class of the source database.

[0039] According to an implementable manner of an embodiment of the present application, the mapping relationship configuration module includes:

[0040] The to-be-synchronized data field selection unit is used to select the source database table in the source database as the original data field;

[0041] The target data field selection unit is used to select the target database table in the target database as the target data field;

[0042] The mapping relationship configuration unit is used to configure the mapping relationship of at least some fields from the source database table to the target database table.

[0043] According to an implementable manner of an embodiment of the present application, the to-be-synchronized data field selection unit is specifically configured to:

[0044] Select the source database and the database table name therein as the source database table;

[0045] Write the pre-sql statement and post-sql statement of the to-be-synchronized data field in the source database table to determine the original data field in the source database table.

[0046] According to an implementable manner of an embodiment of the present application, the task script construction module is specifically configured to:

[0047] Determine the reading tool of the source database and the writing tool of the target database table;

[0048] Write the connection information of the reading tool and the writing tool, the source database table to be synchronized and its pre-sql statement and post-sql statement, the target database table and the target data field into a configuration file;

[0049] Use a python script to read the configuration file and automatically generate a JSON file as the synchronization task script.

[0050] According to an implementable manner of an embodiment of the present application, the data heterogeneous synchronization management device further includes:

[0051] A timing scheduling setting module, configured to set a timing schedule for the data synchronization job task after sending the data synchronization job task created according to the synchronization task script to a job scheduling engine, so that the data synchronization job task is executed according to a set time;

[0052] A job execution log acquisition module, configured to acquire a job execution log, determine the execution status of the data synchronization job task, and display it to a user.

[0053] According to another aspect of the embodiments of the present application, there is provided a computer-readable storage medium, where the computer-readable storage medium includes a stored program, and when the program runs, it executes the data heterogeneous synchronization management method provided in any embodiment of the present application.

[0054] According to another aspect of the embodiments of the present application, there is provided an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is configured to execute the data heterogeneous synchronization management method provided in any embodiment of the present application through the computer program.

[0055] A data heterogeneous synchronization management method, device, storage medium and electronic device provided by the embodiments of the present application, by a user selecting synchronization data fields of a source database and a target database to be synchronized, configuring a mapping relationship of the data fields, and sending a synchronization task script of the data synchronization job task to a server of a job scheduling engine, so that the job scheduling engine performs a data synchronization operation, solves the problems of low efficiency and long cycle of repetitive development jobs, and is insufficient to cope with rapidly iterative requirement development tasks. The data synchronization realizes the visualization of the configuration and management of the data synchronization job task and the convenience of user operation, improves the efficiency and accuracy of data synchronization, and reduces the communication cost of backend interface development and data access development. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0057] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other accompanying drawings can also be obtained based on these drawings without creative efforts.

[0058] Figure 1 is a schematic diagram of the hardware environment of a data heterogeneous synchronization management method of an intelligent device according to an embodiment of the present application;

[0059] Figure 2It is a flowchart of a data heterogeneous synchronization management method according to an embodiment of the present application;

[0060] Figure 3 It is a schematic diagram of a data source configuration interface according to an embodiment of the present application;

[0061] Figure 4 It is a schematic diagram of a selection interface for data fields to be synchronized according to an embodiment of the present application;

[0062] Figure 5 It is a schematic structural diagram of a data heterogeneous synchronization management device according to an embodiment of the present application;

[0063] Figure 6 It is a schematic structural diagram of an electronic device according to an embodiment of the present application. Detailed implementation manners

[0064] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0065] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0066] According to one aspect of the embodiments of the present application, a data heterogeneous synchronization management method for smart home devices is provided. This data heterogeneous synchronization management method is widely applied to the scenario of the digital application system of an online mall, for the situation of data synchronization between databases of multiple data centers for cross-business data. Optionally, in this embodiment, the above-mentioned data heterogeneous synchronization management method for smart home devices can be applied to, for example, Figure 1 the hardware environment composed of a terminal device 102 and a server 104 as shown. As Figure 1As shown, the server 104 is connected to the terminal device 102 through a network and can be used to provide services (such as application services, etc.) for the terminal or the client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data operation services for the server 104.

[0067] The above network can include but is not limited to at least one of the following: wired network, wireless network. The above wired network can include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network. The above wireless network can include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 is not limited to a PC, mobile phone, tablet computer, smart air conditioner, smart range hood, smart refrigerator, smart oven, smart stove, smart washing machine, smart water heater, smart washing equipment, smart dishwasher, smart projection device, smart TV, smart clothes hanger, smart curtain, smart audio and video, smart socket, smart speaker, smart sound box, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart floor sweeping robot, smart window cleaning robot, smart mopping robot, smart air purification equipment, smart steam box, smart microwave oven, smart kitchen water heater, smart purifier, smart water dispenser, smart door lock, etc.

[0068] According to one aspect of the embodiments of the present application, a data heterogeneous synchronization management method is provided, such as Figure 2 As shown in the flowchart of a data heterogeneous synchronization management method, it can be applied to user terminal devices, such as computer devices like personal computers and tablet computers. The data heterogeneous synchronization management method includes:

[0069] Step S220: Obtain the source database to be synchronized from the user-maintained data source list.

[0070] Among them, the user in the embodiment of the present application generally refers to the development management personnel of the online mall business. As development management personnel, they will log in to the business management system on user terminal devices such as their work computers to configure data synchronization tasks. Different services may be deployed in different databases, and there may also be situations where data needs to be synchronized between different services. Usually, a development management personnel will maintain the synchronization work of multiple databases. The user maintenance data source list is the databases that each development management personnel has data synchronization management according to their management authority. The user can obtain the source database to be synchronized from their user maintenance data source list. The user maintenance data source list can display the source databases in the list for the user to select in response to the user's dropdown operation, or can also display the source databases for the user to select according to the name and / or type of the source database selected by the user. The source databases in the user maintenance data source list will include information such as the name, type, and data source grouping of the database, and can also include information such as the username, password, and interface of the source database.

[0071] Step S240: Select the original data fields in the source database and the target data fields in the target database, and configure the mapping relationship of the data fields.

[0072] Among them, the data required by different services generally varies. The data that needs to be synchronized is the common data across services, such as the store data of the online mall. Therefore, the data that needs to be synchronized when the user maintains the database is relatively limited. Only when new, modified, or deleted data that needs to be synchronized across services appears will synchronization processing be required. This generally requires the user to confirm the source database and the target database, and then further determine which data fields in the source database need to be synchronized to the target database, and use them as the original data fields. Usually, the database will use methods such as table partitioning to store data for easy query. The user selects the corresponding data source and uses visual configuration to map the source and target data fields to be synchronized. In this way, when data changes such as new, modification, and deletion occur, the relevant changed data can be synchronized to the target data fields of the target database.

[0073] Step S260: According to the preset data synchronization rules, construct a synchronization task script based on the mapping relationship.

[0074] Among them, the execution of data synchronization generally requires pre-arranging a data synchronization architecture on relevant servers, such as DataX, which is an offline synchronization tool for heterogeneous data sources open-sourced by Alibaba. Its application implementation includes stable and efficient data synchronization functions between various heterogeneous data sources such as relational databases (MSQL, Oracle, etc.), HDFS, Hive, ODPS, HBase, and FTP. According to the configuration rules of the scripts required for data synchronization by the preset data synchronization framework plug-in, a synchronization task script is constructed based on the mapping relationship determined by the above selection. The synchronization task script generally includes the relevant configurations of the source database and the target database.

[0075] Step S280: Create a data synchronization job task according to the synchronization task script and send it to the job scheduling engine so that the job scheduling engine can perform data synchronization operations.

[0076] Among them, when the user completes the construction of the synchronization task script using the visual operation interface of the system, it is considered to have constructed a data synchronization job task. Such a synchronization task script can be sent to the job scheduling engine after verifying the permissions according to the user's username and password. In this way, the preset data synchronization framework plug-in will perform corresponding data synchronization operations by the job scheduling engine. Thus, the selected data fields in the source database will be automatically synchronized to the target database. The data synchronization job task can be published to the job scheduling engine with one click through the visual interface to start the data synchronization job. The user can also set timed scheduling through the job management module. Thus, for data that needs to be synchronized periodically, data synchronization operations can be performed at the set time points. In addition, the unpublished data synchronization job tasks can be edited again by the user as needed.

[0077] By the user selecting the synchronization data fields of the source database and the target database to be synchronized, and configuring the mapping relationship of the data fields, and sending the synchronization task script of the data synchronization job task to the server of the job scheduling engine so that the job scheduling engine can perform data synchronization operations, the data synchronization realizes the visualization of the configuration and management of the data synchronization job task, the convenience of user operation, improves the efficiency and accuracy of data synchronization, and reduces the communication cost of data access development for back-end interface development.

[0078] According to an implementable manner of the embodiment of the present application, before obtaining the source database to be synchronized from the user-maintained data source list, it further includes:

[0079] Configure the data source parameter information of the source database to generate a maintained data source project;

[0080] Add the maintained data source project to the user-maintained data source list under the user account.

[0081] Among them, the user can perform operations such as adding and deleting the data sources maintained by himself. The parameter information of multiple types of data sources can be pre-configured, including URL, account password, etc., to construct a list of data sources maintained by the user.

[0082] According to an implementable manner of an embodiment of the present application, configuring the data source parameter information of the source database to generate a data source item for maintenance includes:

[0083] Configuring the data source type, data source name, user account, user password, database uniform resource locator URL, and database JDBC driver class of the source database.

[0084] Among them, for the configuration of the data source, the data source type, data source name, user account, user password, database uniform resource locator URL, database JDBC driver class, etc. can be set. For example, Figure 3 in the interface for adding a data source shown, the user can conveniently configure the relevant information of the data source. It can be understood that Figure 3 this is only an example of configuring the data source and does not limit the solution of the present application.

[0085] According to an implementable manner of an embodiment of the present application, selecting the data fields to be synchronized in the source database and the target data fields in the target database, and configuring the mapping relationship of the data fields includes:

[0086] Selecting the source database table in the source database as the original data field;

[0087] Selecting the target database table in the target database as the target data field;

[0088] Configuring the mapping relationship of at least some fields from the source database table to the target database table.

[0089] Among them, the data fields in the database can be queried and located through the database name and the table name of the database, and the data fields to be synchronized can be determined specifically through the preamble and postscript statements. For the target data fields, they can also be determined through the partition data fields of the database.

[0090] According to an implementable manner of an embodiment of the present application, selecting the source database table in the source database as the original data field includes:

[0091] Selecting the source database and the database table name serving as the source database table therein;

[0092] Writing the preamble sql statement and the postscript sql statement of the data fields to be synchronized in the source database table to determine the original data fields in the source database table.

[0093] Among them, as Figure 4 shown in the selection interface of the data fields to be synchronized during the construction script process, the user can select the source database in the drop-down list of the database source, then select the database table name in the drop-down list of the database table, and then further write the pre-SQL statement and post-SQL statement of the data fields to be synchronized to determine the original data fields to be synchronized in the source-side database table.

[0094] According to an implementable manner of an embodiment of the present application, constructing a synchronization task script according to the script rules of the preset data synchronization framework plug-in based on the mapping relationship includes:

[0095] Determine the reading tool for the source database and the writing tool for the target-side database table;

[0096] Write the connection information of the reading tool and the writing tool, the source-side database table to be synchronized and its pre-SQL statement and post-SQL statement, the target-side database table and the target data fields into the configuration file;

[0097] Use a Python script to read the configuration file and automatically generate a JSON file as the synchronization task script.

[0098] Among them, first prepare a configuration file, such as datax.ini, and write the connection information of the reading tool reader and the writing tool writer, the tables, fields and their pre-statements to be synchronized in the configuration file. Then, use a Python script to read the configuration file and automatically generate the JSON file config.json of DataX. This json file is the job task of DataX and can directly run to synchronize data.

[0099] According to an implementable manner of an embodiment of the present application, after creating a data synchronization job task according to the synchronization task script and sending it to the job scheduling engine, it further includes:

[0100] Set a timed schedule for the data synchronization job task so that the data synchronization job task is executed according to the set time;

[0101] Obtain the job execution log, determine the execution situation of the data synchronization job task and display it to the user.

[0102] Among them, the management system can support custom plug-in extensions and flexible synchronization policy configurations to adapt to different data synchronization requirements. It can also monitor the execution status of synchronization tasks in real time and provide detailed log information for easy troubleshooting and operation and maintenance management. The configuration, monitoring, and management of data synchronization tasks are intuitive and convenient, improving the efficiency of data synchronization, reducing the communication costs of data access development for backend interface development, and providing an efficient, easy-to-use, and flexible solution for data synchronization and mutual access.

[0103] According to another aspect of the embodiments of the present application, a data heterogeneous synchronization management device is provided. As shown in the schematic structural diagram of the data heterogeneous synchronization management device Figure 5 configured in the user terminal device, the data heterogeneous synchronization management device includes:

[0104] A source database acquisition module 502, configured to acquire the source database to be synchronized from the user-maintained data source list;

[0105] A mapping relationship configuration module 504, configured to select the original data fields in the source database and the target data fields in the target database, and configure the mapping relationship of the data fields;

[0106] A task script construction module 506, configured to construct a synchronization task script according to the preset data synchronization rules and the mapping relationship;

[0107] A synchronization job task sending module 508, configured to create a data synchronization job task according to the synchronization task script and send it to the job scheduling engine for the job scheduling engine to perform data synchronization operations.

[0108] According to an implementable manner of the embodiments of the present application, the data heterogeneous synchronization management device further includes:

[0109] A maintenance data source project generation module, configured to configure the data source parameter information of the source database and generate a maintenance data source project before acquiring the source database to be synchronized from the user-maintained data source list;

[0110] A user-maintained data source list determination module, configured to add the maintenance data source project to the user-maintained data source list under the user account.

[0111] According to an implementable manner of the embodiments of the present application, the maintenance data source project generation module is specifically configured to:

[0112] Configure the data source type, data source name, user account, user password, database uniform resource locator URL, and database JDBC driver class of the source database.

[0113] According to an implementable manner of an embodiment of the present application, the mapping relationship configuration module 504 includes:

[0114] A to-be-synchronized data field selection unit, configured to select a source database table in the source database as the original data field;

[0115] A target data field selection unit, configured to select a target database table in the target database as the target data field;

[0116] A mapping relationship configuration unit, configured to configure a mapping relationship between at least some fields of the source database table and the target database table.

[0117] According to an implementable manner of an embodiment of the present application, the to-be-synchronized data field selection unit is specifically configured to:

[0118] Select the source database and the database table name serving as the source database table therein;

[0119] Write a pre-SQL statement and a post-SQL statement of the to-be-synchronized data field in the source database table to determine the original data field in the source database table.

[0120] Wherein, a user can select the source database in a drop-down list of the database source, then select the database table name in a drop-down list of the database table name, and then further write the pre-SQL statement and the post-SQL statement of the to-be-synchronized data field to determine the data field to be synchronized in the source database table.

[0121] According to an implementable manner of an embodiment of the present application, the task script construction module 506 is specifically configured to:

[0122] Determine a reading tool for the source database and a writing tool for the target database table;

[0123] Write connection information of the reading tool and the writing tool, the source database table to be synchronized and its pre-SQL statement and post-SQL statement, the target database table and the target data field into a configuration file;

[0124] Use a Python script to read the configuration file and automatically generate a JSON file as the synchronization task script.

[0125] Among them, first prepare a configuration file, such as datax.ini, and write the connection information of the reader and writer, the tables to be synchronized, fields, and their preamble statements, etc. in the configuration file. Then, use a Python script to read the configuration file and automatically generate the JSON file config.json of DataX. This JSON file is the job task of DataX and can directly run to synchronize data.

[0126] According to an implementable manner of an embodiment of the present application, the data heterogeneous synchronization management device further includes:

[0127] A timing scheduling setting module, configured to set timing scheduling for the data synchronization job task after sending the data synchronization job task created according to the synchronization task script to the job scheduling engine, so that the data synchronization job task is executed according to the set time;

[0128] A job execution log acquisition module, configured to acquire job execution logs, determine the execution status of the data synchronization job task, and display it to the user.

[0129] Among them, the management system can support custom plugin extension and flexible synchronization policy configuration, adapt to different data synchronization requirements, and can monitor the execution status of synchronization tasks in real time, provide detailed log information, facilitate problem troubleshooting and operation and maintenance management. The configuration, monitoring, and management of data synchronization tasks are intuitive and convenient, improving the efficiency of data synchronization, reducing the communication cost of data access development for backend interface development, and providing an efficient, easy-to-use, and flexible solution for data synchronization and inter-access.

[0130] According to another aspect of an embodiment of the present invention, Figure 6 The structure diagram of an electronic device provided by an embodiment of the present invention is shown in Figure 6 As shown, the electronic device includes a processor 610, a memory 620, an input device 630, and an output device 640; the number of processors 610 in the electronic device can be one or more, Figure 6 Taking one processor 610 as an example; the processor 610, memory 620, input device 630, and output device 640 in the electronic device can be connected through a bus or other means, Figure 6 Taking connection through a bus as an example.

[0131] The memory 620, being a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the data heterogeneous synchronization management method in the embodiments of the present invention (for example, the source database acquisition module 502, the mapping relationship configuration module 504, the task script construction module 506, and the synchronization job task sending module 508 in the data heterogeneous synchronization management device). The processor 610 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 620, that is, implements the above-mentioned data heterogeneous synchronization management method.

[0132] The memory 620 may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory 620 may further include a memory remotely set relative to the processor 610, and these remote memories can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0133] The input device 630 can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the electronic device. It can also be a camera for acquiring images and a sound pickup device for acquiring audio data. The output device 640 may include display devices such as a display screen and may include audio devices such as speakers. It should be noted that the specific composition of the input device 630 and the output device 640 can be set according to actual situations. The processor 610 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 620.

[0134] According to another aspect of the embodiments of the present invention, the embodiments of the present invention also provide a computer-readable storage medium. The computer-readable storage medium includes a stored program, wherein when the program runs, it executes the following data heterogeneous synchronization management method:

[0135] Obtain the source database to be synchronized from the user-maintained data source list;

[0136] Select the original data fields in the source database and the target data fields in the target database, and configure the mapping relationship of the data fields;

[0137] Construct a synchronization task script according to the mapping relationship according to the preset data synchronization rules;

[0138] Create a data synchronization job task according to the synchronization task script and send it to the job scheduling engine so that the job scheduling engine can perform data synchronization operations.

[0139] Of course, for a storage medium containing computer-executable instructions provided by an embodiment of the present invention, the computer-executable instructions are not limited to the method operations described above, and can also execute related operations in the data heterogeneous synchronization management method provided by any embodiment of the present invention.

[0140] From the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software and necessary general hardware. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as a floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk, or an optical disc of a computer, etc., including several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention.

[0141] It should be noted that in the embodiments of the above data heterogeneous synchronization management device, the included units and modules are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the present invention.

[0142] The above are only the preferred embodiments of the present application. It should be pointed out that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A method for managing heterogeneous data synchronization, characterized in that Including: Obtain the source database to be synchronized from the user-maintained data source list; Select the original data fields in the source database and the target data fields in the target database, and configure the mapping relationship of the data fields; Construct a synchronization task script according to the mapping relationship in accordance with the preset data synchronization rules; Create a data synchronization job task according to the synchronization task script and send it to the job scheduling engine so that the job scheduling engine performs data synchronization operations.

2. The data heterogeneous synchronization management method according to claim 1, wherein Before obtaining the source database to be synchronized from the user-maintained data source list, it further includes: Configure the data source parameter information of the source database to generate a maintained data source project; Add the maintained data source project to the user-maintained data source list under the user account.

3. The data heterogeneous synchronization management method according to claim 2, wherein The configuring the data source parameter information of the source database to generate a maintained data source project includes: Configure the data source type, data source name, user account, user password, database uniform resource locator URL, and database JDBC driver class of the source database.

4. The data heterogeneous synchronization management method according to claim 1, wherein The selecting the original data fields in the source database and the target data fields in the target database and configuring the mapping relationship of the data fields includes: Select the source-side database table in the source database as the original data field; Select the target-side database table in the target database as the target data field; Configure the mapping relationship of at least some fields from the source-side database table to the target-side database table.

5. The data heterogeneous synchronization management method according to claim 4, wherein The selecting the source-side database table in the source database as the original data field includes: Select the source database and the database table name that serves as the source-side database table therein; Write the pre-sql statement and post-sql statement of the data fields to be synchronized in the source-side database table to determine the original data fields in the source-side database table.

6. The data heterogeneous synchronization management method according to claim 5, wherein The constructing a synchronization task script according to the mapping relationship in accordance with the preset data synchronization rules includes: Determine the reading tool of the source database and the writing tool of the target-side database table; Write the connection information of the reading tool and the writing tool, the source-side database table to be synchronized and its pre-sql statement and post-sql statement, the target-side database table and the target data fields into a configuration file; Use a python script to read the configuration file and automatically generate a JSON file as the synchronization task script.

7. The data heterogeneous synchronization management method according to claim 1, characterized in that After creating a data synchronization job task according to the synchronization task script and sending it to the job scheduling engine, it further includes: Set a timed schedule for the data synchronization job task so that the data synchronization job task is executed according to the set time; Obtain the job execution log, determine the execution status of the data synchronization job task and display it to the user.

8. A data heterogeneous synchronization management device, characterized in that, Including: A source database acquisition module for obtaining the source database to be synchronized from the user-maintained data source list; A mapping relationship configuration module for selecting the original data fields in the source database and the target data fields in the target database and configuring the mapping relationship of the data fields; A task script construction module for constructing a synchronization task script according to the mapping relationship in accordance with the preset data synchronization rules; The synchronous job task sending module is used to create a data synchronization job task according to the synchronous task script and send it to the job scheduling engine, so that the job scheduling engine can perform data synchronization operations.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program, when running, executes the data heterogeneous synchronization management method according to any one of claims 1 to 7.

10. An electronic device, comprising a memory and a processor, characterized in that, A computer program is stored in the memory, and the processor is configured to execute the data heterogeneous synchronization management method according to any one of claims 1 to 7 through the computer program.

Images