Data synchronization method and apparatus, electronic device, computer-readable storage medium

Abstract

The present disclosure provides a data synchronization method and device, electronic equipment and computer readable storage medium, the method comprising: querying metadata from a source data source through a metadata query interface, and obtaining target data to be synchronized from the metadata; generating and deploying a synchronization configuration file corresponding to the tool type based on the tool type of the selected synchronization tool; and synchronizing the target data to a target data source based on the synchronization configuration file through the synchronization tool. According to the embodiments of the present disclosure, the data synchronization efficiency can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a data synchronization method and apparatus, electronic device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of computer technology and the internet industry, the amount of data generated in all walks of life and in daily life is increasing daily, such as employee information, order records, and web browsing history. Because different databases have different functional characteristics, they can be used to manage data in different ways. To improve data management efficiency and maintain data integrity and consistency, data synchronization between different databases is necessary.

[0003] A common data synchronization method involves writing Structured Query Language (SQL) code using a data synchronization tool, executing the SQL in the data source management tool to retrieve the data to be synchronized, and then writing the synchronized data to the target storage medium according to a specified method. While this method can be automated to some extent using scripting languages, it involves a significant workload for development and testing, resulting in low data synchronization efficiency. Therefore, how to perform efficient data synchronization has become a hot research topic in the field. Summary of the Invention

[0004] This disclosure provides a data synchronization method and apparatus, electronic device, and computer-readable storage medium, which can improve data synchronization efficiency.

[0005] In a first aspect, this disclosure provides a data synchronization method, which includes: obtaining metadata from a source data source through a metadata query interface, and retrieving the target data to be synchronized from the metadata; generating and deploying a synchronization configuration file corresponding to the selected synchronization tool type; and synchronizing the target data to the target data source based on the synchronization configuration file using the synchronization tool.

[0006] Secondly, this disclosure provides a data synchronization device, which includes: a query module for querying metadata from a source data source through a metadata query interface and obtaining target data to be synchronized from the metadata; a configuration module for generating and deploying a synchronization configuration file corresponding to the selected synchronization tool type; and a synchronization module for synchronizing the target data to the target data source through a synchronization tool based on the synchronization configuration file.

[0007] Thirdly, this disclosure provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the data synchronization method described above.

[0008] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-described data synchronization method when executed by a processor / processor core.

[0009] Fifthly, this disclosure provides a computer program or computer program product, the computer program product including a computer program that can be stored in a computer-readable storage medium, and the computer program implementing the above-described data synchronization method when executed by a processor.

[0010] The data synchronization method provided in this disclosure retrieves metadata through a metadata query interface, obtains the target data to be synchronized from the retrieved metadata, and automatically generates synchronization configuration files for different types of synchronization tools, thereby adapting to different types of synchronization tools. The data synchronization process does not require the development and testing of data acquisition scripts, which helps to reduce the workload of data synchronization and improve data synchronization efficiency.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0013] Figure 1 A flowchart of a data synchronization method provided in an embodiment of this disclosure;

[0014] Figure 2 A schematic diagram of the framework of a data synchronization system provided in this embodiment of the disclosure;

[0015] Figure 3 A block diagram of a data synchronization device provided in an embodiment of this disclosure;

[0016] Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0019] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0020] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, they specify the presence of features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0021] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0022] In related technologies, data synchronization methods include the following steps: Developing a data acquisition script based on data synchronization requirements; and, after the data acquisition script passes script testing, creating a corresponding acquisition configuration file according to the acquisition tool's specifications; saving the acquisition configuration file to a specified directory on the server where the acquisition tool is located, and starting the acquisition task; during the execution of the acquisition task, viewing the task output logs and running results, and manually determining if there are any acquisition anomalies; if acquisition anomalies exist, repairing the data acquisition script or configuration file, and restarting the task; if no anomalies exist, the data synchronization is complete.

[0023] In the data synchronization methods of related technologies, the workload of developing and testing data acquisition scripts is relatively large; commonly used data synchronization tools do not support the automatic generation of standardized acquisition configuration files, and can only rely on manual analysis of data synchronization requirements and manual writing of synchronization configuration files, resulting in low data synchronization efficiency.

[0024] According to the data synchronization method of this disclosure, synchronization configuration files for different types of synchronization tools can be automatically generated through metadata, thereby adapting to different types of synchronization tools; and, there is no need to develop and test data acquisition scripts, which helps to reduce the workload of data synchronization and improve data synchronization efficiency.

[0025] The data synchronization method according to embodiments of this disclosure can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The method can be implemented by a processor calling computer-readable program instructions stored in memory. Servers can include independent physical servers, server clusters consisting of multiple servers, or cloud servers capable of cloud computing.

[0026] See Figure 1 This is a flowchart illustrating a data synchronization method provided in an embodiment of this disclosure. (Refer to...) Figure 1 The method includes the following steps:

[0027] S110 retrieves metadata from the source data source through the metadata query interface and obtains the target data that needs to be synchronized from the metadata.

[0028] In some embodiments, the target data to be synchronized is the data to be synchronized; the source data source refers to the data source that provides the data to be synchronized, and the target data source is the data source that needs to be synchronized with the source data source. Data synchronization is a data processing method that synchronizes the data to be synchronized from the source data source to the target data source. Metadata is data used to manage data and can be understood as data used to describe the data structure.

[0029] In some embodiments, the source data source type includes: Relational Database Management System (RDBMS) type and system interface type. In some scenarios, RDBMS type data sources are also referred to as traditional data sources or database type data sources. That is, in other words, the source data source type includes database type and system interface type.

[0030] In some embodiments, metadata includes the data source name, entity information, and corresponding entity attribute information; step S110 may specifically include the following steps:

[0031] S11, call the metadata query interface, use the data source name to query the source data source, and obtain the corresponding multiple candidate entity information and multiple candidate entity attribute information from the found source data source.

[0032] As an example, the metadata query interface is called to query the data source (source data source) that needs to be mapped using the data source name. In the found source data source, multiple entity information and multiple entity attribute information are obtained. The entity information and entity attribute information corresponding to the data source name are used as candidate information.

[0033] S12, display multiple candidate entity information and multiple candidate entity attribute information on the page; wherein, the page contains multiple first control elements for marking multiple candidate entity information and multiple second control elements for marking multiple candidate entity attribute information, one candidate entity information corresponds to one first control element, and one candidate entity attribute information corresponds to one second control element.

[0034] As an example, both the first and second control elements can be selection controls, such as at least one of checkboxes or multi-select labels.

[0035] S13, in response to the operation instructions for multiple first control elements and multiple second control elements, the candidate entity information marked by the operated first control element and the candidate entity attribute information corresponding to the operated second control element are used as target data to be synchronized.

[0036] In this embodiment, the candidate information obtained from the query can be displayed through a web page. The web page provides information selection functionality through a selection control to select entity information and entity attribute information from the displayed candidate information. Based on the selected entity information and selected entity attribute information on the web page, the target data to be synchronized (i.e., the data to be synchronized) can be determined.

[0037] In this embodiment of the disclosure, entity information and entity attribute information to be synchronized are selected from metadata via the web, and a synchronization configuration file can be automatically generated according to the type of synchronization tool. This not only reduces the configuration difficulty, but also helps to reduce errors that are easily caused when manually writing data query statements, thereby reducing the error rate of data collection and improving the accuracy of data collection.

[0038] In this embodiment of the disclosure, metadata can be extracted from data sources of database type and system interface type, and the information of the metadata can be abstracted and summarized, dividing the information in the metadata into three levels: data source name, entity information, and entity attribute information. For example, the information model of the metadata can be constructed from the three levels of data source name, entity, and attribute, so as to manage the metadata from the three levels of data source name, entity, and attribute.

[0039] As mentioned above, the source data type includes database type and system interface type. If the source data type is database type, then the data source name is the database name, the entity information is the table name, and the entity attribute information is the field name in the corresponding table. For example, if the source data source is a MySQL database, the data source name is the database name, the entity information is the table name, the entity attribute information is the table field, and the attribute value is the field content.

[0040] If the source data source is a system interface, then the data source name is the interface name, the entity information is the request and response structure information of the system interface, and the entity attribute information is the field information in the corresponding request structure and the corresponding response structure. Field information includes the field name and the field value. For example, if the source data source is a system interface, then the data source name is the interface name, the entity information is the response and request structure of that interface, and the entity attribute information is the fields in the request and response structures; attribute values ​​are the field content.

[0041] In some embodiments, each entity attribute has a specific type. For example, when the source data source type is a database type, the attribute type can be a field type of a table field; when the source data source type is a system interface type, the attribute type can be a field type of fields contained in the request structure and response structure. Exemplarily, the field type can be at least one of the following types: binary type, character type, date and time type, numeric type, currency type, etc. It should be understood that data types can be customized according to actual circumstances, and this disclosure does not impose specific limitations.

[0042] For ease of understanding, Table 1 below describes the specific content of the system interface metadata when the source data source type is a system interface type.

[0043] Table 1. Detailed content of system interface metadata

[0044]

[0045] In Table 1, the system interface is named "Get User Information," indicating that this system interface is used to obtain user information. Requested interface information may include, for example, Position Id, Position No, Employee No., Employee Name, and Department No. The response structure information may include, for example, Employee No., Employee Id, and Employee Basic Information (BaseInfo). The Employee Basic Information includes at least one of the following items: Company Email, Age, Political Affiliation, Job Years, Job Grade, Highest Education Level (Maxedudiploma), Highest Education Major (Maxeduspecialty), AvatarPath, Department Id, Employee Position Name, Marital Status, Gender, Labor Relationship Status, and Employee Job Name.

[0046] For example, department names can also include more detailed department information, such as the name of a third department (Deptthreename), a fourth department (Deptfourname), a fifth department (Deptfivename), a sixth department (Deptsixname), etc., belonging to a certain headquarters. For example: R&D Department 1, R&D Department 2, ..., R&D Department 6, etc.

[0047] In Table 1, when the source data type is a system interface type, the data source name is the interface name, the entity information is the request structure information and response structure information of the system interface, and the entity attribute information is the field information in the corresponding request structure and the field information in the corresponding response structure; related attribute information may be, for example, interface link information.

[0048] It should be understood that the specific content of the request structure information and response structure information in Table 1 is merely illustrative, and the specific content can be determined according to the actual situation. Furthermore, the request structure information and response structure information of this system interface may also include more different information items, which can be customized according to actual needs. This disclosure embodiment does not impose specific limitations.

[0049] In the data synchronization method of this disclosure, both traditional data synchronization with database as the source data source and data synchronization with system interface as the source data source are supported. Compared with related technologies, data synchronization tools only support database-type data synchronization and do not support interface-type data synchronization. The data synchronization method of this disclosure supports interface-type data synchronization, which expands the application scenarios of data synchronization and can meet the data synchronization needs of more different types of data sources.

[0050] In this embodiment of the disclosure, different types of source data sources may have their own unique attributes; in some embodiments, the metadata also includes: relevant attribute information corresponding to the data source type. When the source data source type is a database type, the relevant attribute information includes at least one of the following information items: database connection information and database login information; when the source data source type is a system interface type, the relevant attribute information includes at least one of the following information items: interface connection information and interface configuration information.

[0051] For example, when the source data source is a MySQL database, the relevant attribute information includes the database connection (UniformResource Locator, URL) and login information such as username and password required to log in to the database; when the source data source is a system interface, the relevant attribute information may include interface configurations such as timeout control and flow control restrictions; it should be understood that the relevant attribute information of different types of source data sources can be customized according to the actual situation, and this disclosure embodiment does not make specific limitations.

[0052] In the data synchronization method of this disclosure embodiment, for different types of source data sources, in addition to dividing the information in the metadata into three levels: data source name, entity information, and entity attribute information, and managing the metadata from these three levels, the method can also manage the unique attribute information of different types of source data sources, thereby realizing data synchronization of diverse metadata from different types of data sources.

[0053] S120 generates and deploys a synchronization configuration file corresponding to the selected synchronization tool type, based on the tool type of the selected synchronization tool.

[0054] In some embodiments, the data synchronization method of this disclosure can be adapted to various synchronization tools. The tool type in this step is used to indicate different synchronization tools. As an example, the synchronization tools include, but are not limited to, at least one of the following types: offline data synchronization tool / platform DataX, distributed offline and real-time data synchronization tool FlinkX, change data capture component based on a distributed computing engine (Flink Change Data Capture, FlinkCDC), and data warehouse tool Sqoop.

[0055] Among them, DataX is an open-source offline synchronization tool for heterogeneous data sources, enabling stable and efficient data synchronization between various heterogeneous data sources, including relational databases (MySQL, Oracle, etc.), distributed file systems (Hadoop Distributed File System, HDFS), data warehouse tools (Hive), Ontology Design Patterns (ODPS), column-oriented distributed databases (HBase), and file transfer services (FTP). FlinkX is a distributed offline / real-time data synchronization plugin based on Flink, enabling efficient data synchronization across multiple heterogeneous data sources. Flinkcdc is a resource component that can directly read full data and incrementally changed data from relational databases; through Change Data Capture (CDC) technology, it synchronizes incremental change records from the source database to one or more destinations. Sqoop is a data transfer tool based on the Apache server software, used for data transfer between Hadoop, Hive, and traditional relational databases. For example, Sqoop can import data from a relational database into Hadoop's HDFS, and vice versa.

[0056] It should be understood that the synchronization tools in the embodiments of this disclosure may also have other different types; the specific types can be selected and adjusted according to actual needs, and the embodiments of this disclosure do not impose specific limitations.

[0057] In some embodiments, step S120 may specifically include the following steps.

[0058] S21, obtain the selected index information of the target data source, and call the metadata query interface to use the index information to query the entity attribute information corresponding to the index information from the metadata.

[0059] The target data source in this disclosure can also be of various types, such as relational databases, APIs, or file management systems. For example, the target data source can be a distributed open-source search and analytics engine (Elasticsearch, ES), which can be used for data collection, storage, search, analysis, and visualization management. ES is applicable to various types of data, including text, numbers, structured data, and unstructured data.

[0060] In some embodiments, taking Elasticsearch (ES) as the target data source, a document in ES is the smallest unit of all searchable data, similar to a row in a table in a relational database. Each document in ES has a unique identifier, which can be customized or automatically generated by the system. Index information is used to index entity attribute information in the target data source. Through the index of ES, a fast full-text search can be performed in ES to obtain the entity attribute information in ES corresponding to the index. Compared with database-type and interface-type data sources, ES is more suitable for relevance and high-performance full-text retrieval. Therefore, in some application scenarios of this disclosure embodiment, it is necessary to use ES as the target data source for data synchronization.

[0061] S22, determine the one-to-one correspondence between the entity attribute information in the metadata and the entity attribute information corresponding to the index information.

[0062] In this step, during data synchronization, the values ​​of entity attribute information in the metadata of the data source (such as the values ​​of table fields in the database) need to be written to the entity attribute information (such as the corresponding field information) under a certain index information of the target data source. Therefore, when synchronizing data between the source data source and the target data source, it is necessary to determine the one-to-one correspondence between the entity attribute information in the metadata (entity attribute information in the source data source) and the entity attribute information corresponding to the index information (entity attribute information in the target data source).

[0063] S23. Based on the tool type of the synchronization tool, obtain the synchronization configuration parameters corresponding to the tool type.

[0064] This step generates synchronization configuration parameters for different types of synchronization tools. These configuration parameters can be implemented using a scripting language, resulting in synchronization configuration scripts for different types of synchronization tools. A script is an executable file written in a specific descriptive language according to a certain format; it is also known as a macro or batch file. Scripts can typically be temporarily called and executed by applications.

[0065] S24. Based on the one-to-one correspondence and synchronization configuration parameters, generate a synchronization configuration file corresponding to the tool type, which is used to record the synchronization correspondence between entity attribute information in the target data and entity attribute information corresponding to index information through the synchronization configuration file.

[0066] In this embodiment, compared to related technologies, data synchronization tools do not support the automatic generation of standardized collection configuration files, and can only be analyzed manually or the configuration files must be written manually according to the specifications of the collection tool; the data synchronization method of this embodiment can automatically generate synchronization configuration information that conforms to the specifications of the selected synchronization tool according to the specific different types of synchronization tools.

[0067] In this embodiment of the disclosure, after obtaining the target data to be synchronized from the metadata, a corresponding synchronization configuration file can be generated according to the type of synchronization tool, and the synchronization configuration file can be deployed.

[0068] In some embodiments, after step S24, the method further includes: S25, deploying the synchronization configuration file to a specified file path; wherein the synchronization tool has access permissions to the specified file path.

[0069] In this step, after the configuration file is generated, it can be deployed to a directory accessible to the synchronization tool. This allows the synchronization tool to call the synchronization configuration file during data synchronization and synchronize the target data based on the correspondence between the entity attribute information in the source data source and the entity attribute information in the target data source recorded in the synchronization configuration file, as well as the synchronization configuration parameters recorded in the synchronization configuration file.

[0070] In this embodiment of the disclosure, the synchronization configuration file is deployed to a path accessible to the selected synchronization tool, thereby enabling unified management of synchronization configuration information, automatic generation and unified deployment of synchronization configuration files, improving the generation efficiency of data synchronization information, and facilitating the automated management of synchronization configuration information and improving data synchronization efficiency.

[0071] In some embodiments, step S25, deploying the synchronization configuration file to a specified file path, may specifically include: mounting the synchronization configuration file to a local file directory using a shared file approach, so as to allow shared access to the synchronization configuration file locally. In this embodiment, a synchronization tool can be used to synchronize target data to a target data source based on the synchronization configuration file. Specifically, this step may include: starting multiple synchronization instances of a pre-created synchronization tool; wherein each synchronization instance is used to synchronize the target data; the synchronization configuration file is shared locally by multiple synchronization instances, and the target data is synchronized to the target data source by multiple synchronization instances working together based on the synchronization configuration file.

[0072] In this embodiment, the shared path can be implemented through shared mounting. For example, the shared path can be a mount directory for a specified file path. The synchronization configuration file is stored in the specified file path using Network File System (NFS) shared storage, and then mounted to the local device via shared mounting. Multiple instances of the synchronization tool run on the local device, allowing the multiple instances of the synchronization tool to read the synchronization configuration file for data synchronization, thus improving data synchronization efficiency.

[0073] S130 uses a synchronization tool to synchronize target data to the target data source based on a synchronization configuration file.

[0074] In this embodiment of the disclosure, after obtaining metadata from the source data source through the metadata query interface, the target data to be synchronized can be selected from the metadata. Furthermore, a corresponding synchronization configuration file can be generated and deployed based on the type of the selected synchronization tool. The target data can then be synchronized from the source data source to the target data source using the synchronization tool and the synchronization configuration file. This data synchronization method can automatically generate synchronization configuration files for different types of synchronization tools through metadata, thus adapting to different types of synchronization tools. Moreover, it eliminates the need to develop and test data acquisition scripts, thereby reducing the workload of data synchronization and improving data synchronization efficiency.

[0075] In some embodiments, step S130 may specifically include the following steps.

[0076] S31, Generate a synchronization task based on the received task scheduling expression.

[0077] As an example, a task scheduling expression can be a cron expression. A cron task is a pre-planned task that executes work at a scheduled time. For example, you can specify in the synchronization configuration file that data synchronization should start at 3 AM every day; starting data synchronization at 3 AM every day is a cron task. A cron expression is an expression used to indicate when a task will be executed; it is typically used to configure the trigger time of a cron task. A cron expression can be a string consisting of six or seven sub-expressions (fields), separated by spaces. For example, a typical cron expression field includes: seconds, minutes, hours, days, weeks, months, and years. By filling in the received cron scheduling expression, the scheduled task can be created.

[0078] S32, configure task scheduling information according to the type of source data and tool type.

[0079] As an example, task scheduling information can be configured based on different types of data sources and different types of synchronization tools, so that data synchronization tasks can be scheduled based on this task scheduling information.

[0080] S33 schedules synchronous tasks based on task scheduling information.

[0081] S34, when the scheduled synchronization task is in the started state, the synchronization tool synchronizes the target data to the target data source according to the synchronization configuration file.

[0082] Through the above steps S31-S34, the synchronization task is scheduled according to the configured task scheduling expression. When the task scheduling starts, the local synchronization tool is started according to the configuration file to synchronize the target data to the target data source, so as to synchronize the target data that needs to be synchronized in the source data source of the corresponding type to the target data source, thereby realizing the automatic scheduling of the data synchronization task.

[0083] In some embodiments, step S32 may specifically include: S41, determining the task execution mode supported by the synchronization tool according to the tool type, and setting the task execution parameters corresponding to the task execution mode according to the source data source type; S42, generating task scheduling information corresponding to the task execution parameters.

[0084] In this embodiment, if the synchronization tool supports distributed task execution, a distributed task scheduling platform can be used to schedule the synchronization tasks. For example, this distributed task scheduling platform can be XXL-JOB, an open-source, lightweight distributed task scheduling platform with rich task management functions and high performance and high availability. If the synchronization tool does not support distributed task execution, a data sharding-based task execution method can be used to schedule the synchronization tasks.

[0085] It should be understood that the task execution methods supported by different types of synchronization tools can be obtained in advance, thereby enabling flexible scheduling of synchronization tasks based on the task execution methods supported by the type of synchronization tool.

[0086] In some embodiments, the task execution method includes a task execution method based on data sharding; step S41, which sets the task execution parameters corresponding to the task execution method according to the type of the source data source, may specifically include the following steps.

[0087] S51, when the source data type is a database, the pagination parameter is used as the task execution parameter for the data sharding-based task execution method; wherein, the pagination parameter is used to indicate that the target data is sharded by pagination.

[0088] For example, if the open-source version of a synchronization tool such as DataX does not support distributed systems, then sharding is performed by configuring sharding parameters. The sharding parameters can include the primary key ID of the data. The data is split according to the primary key ID to determine the data range of each shard.

[0089] S52, when the source data type is a system interface type, the data row identifier and data row creation time are used as task execution parameters for the data sharding-based task execution mode; wherein, the data row identifier and data row creation time are used to indicate that the target data is sharded according to the data row identifier and data row creation time.

[0090] For example, if the synchronization tool supports a data sharding-based task execution method and the source data source is a system interface type, the sharding parameters may include the primary key ID of the data row and the data row creation time; the data is then split according to the primary key ID and data row creation time of each row of data, thereby determining the data range of each shard.

[0091] In this embodiment, the configuration scheduling task and configuration sharding parameters can be specifically set according to the task execution method supported by the synchronization tool and the type of source data source, thereby enabling automatic adaptation with different types of synchronization tools and different types of source data sources, which is beneficial to improving the flexibility of data synchronization methods; and, by customizing sharding, the problem that some open-source versions of synchronization tools do not support distributed processing can be compensated for. Synchronization tools are usually single-machine versions. Data sharding enables single-machine version synchronization tools to achieve distributed job processing, thereby improving synchronization performance.

[0092] In some embodiments, the synchronization configuration file is used to record the synchronization correspondence between entity attribute information in the target data and entity attribute information in the target data source; when the scheduled synchronization task is in the started state, the step of synchronizing the target data to the target data source by the synchronization tool according to the synchronization configuration file in step S34 may specifically include: when the scheduled synchronization task is in the started state, generating and sending a synchronization command to the proxy server according to the type of synchronization tool.

[0093] The proxy server is used to start the synchronization tool according to the synchronization command, obtain the synchronization correspondence through the synchronization tool, and execute task scheduling information to write the target data to the target data source according to the synchronization correspondence.

[0094] In this embodiment, the agent server provides agent services. In some embodiments, an agent program is installed on the server where the synchronization tool is deployed to provide agent services. The data synchronization system of this disclosure embodiment can interact with the synchronization tool through the agent service to send control signals and log collection instructions to the synchronization tool. In this disclosure embodiment, when a synchronization task is started, a task start command is sent to the agent server. After receiving the synchronization task start command, the agent server establishes a connection with the synchronization tool through the agent service. The synchronization tool obtains the synchronization configuration file, acquires the target data according to the synchronization configuration file, and writes the target data to the target data source. By using an agent-based task scheduling framework, the time efficiency of task execution can be improved and the network load can be reduced.

[0095] In some embodiments, the data synchronization method further includes: S61, during the process of synchronizing target data to the target data source based on the synchronization configuration file, collecting logs of the synchronization process through a proxy server; S62, sending the collected log content to the data interface of the synchronization tool so that the log content can be displayed on a page through the synchronization tool.

[0096] In this embodiment, an Agent service monitors log files and uploads the log content to the synchronization platform for real-time display. During data synchronization, outputting logs to the web interface facilitates viewing and identifying synchronization errors. Based on pre-configured synchronization exception policies, tasks can be automatically retried or stopped according to the error type, or tasks can be manually retried or stopped. This enables web-based management of the data synchronization process and provides functions such as automatic retry and automatic scheduling.

[0097] According to the data synchronization method of this disclosure, synchronization configuration files for different types of synchronization tools can be automatically generated through metadata, thereby adapting to different types of synchronization tools. This eliminates the need to develop and test data collection scripts, reducing the workload of data synchronization and improving data synchronization efficiency. The synchronization tools used in the data synchronization method of this disclosure can support interface-type data synchronization, thus expanding the application scenarios of data synchronization and meeting the data synchronization needs of more different types of data sources. Furthermore, the synchronization tools support the automatic generation of synchronization configuration files corresponding to the type of synchronization tool, facilitating the standardized writing and management of collection configuration files for different types of synchronization tools. Finally, the data synchronization method of this disclosure supports web-based management, enabling automatic retries and automatic scheduling during the data synchronization process, thereby improving the efficiency and performance of the data synchronization method.

[0098] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0099] Based on the above data synchronization method, this application provides a data synchronization system, see [link to relevant documentation]. Figure 2 A schematic diagram of the framework of a data synchronization system according to an exemplary embodiment of the present disclosure is shown.

[0100] In this embodiment, the data synchronization system can be implemented using a development framework (Spring Boot). Spring Boot is a browser / server (B / S) framework and an essential framework for developing with the Java programming language technology stack. Spring Boot simplifies the web service development process.

[0101] exist Figure 2 The architecture of this data synchronization system includes a metadata management subsystem 10 and a synchronization platform subsystem 20.

[0102] like Figure 2 As shown, the metadata management subsystem 10 includes a database metadata management module 11, an interface type metadata management module 12, and a metadata acquisition module 13; the synchronization platform subsystem 20 includes a mapping configuration module 21, a mapping file creation module 22, a file deployment module 23, a task scheduling module 24, a task startup module 25, and a viewing module 26.

[0103] In some embodiments, the metadata management subsystem 10 is used to manage metadata in one or more preset types of source data sources. The types of source data sources include, for example, database types and system interface types; the database metadata management module 11 is used to manage metadata in database-type source data sources, and the interface metadata management module 12 is used to manage metadata in system interface-type source data sources.

[0104] For example, metadata includes the data source name, entity information, and corresponding entity attribute information. For database-type source data sources, the data source name is the database name, the entity information is the table name, and the corresponding entity attribute information is the column name with the attribute type being the field type. For system interface-type source data sources, the data source name is the interface name, the entity is the response and request structures of the interface, and the entity attribute information is the field information in the corresponding request structure and the field information in the corresponding response structure with the attribute type being the field type. In some embodiments, view objects (VOs) can be used for data transfer between business layers, and the interface name of the business layer can be represented by the name of the corresponding view object.

[0105] The following is combined with Figure 2 The data synchronization system shown includes various modules, and a data synchronization method according to an exemplary embodiment of this disclosure is described. In some embodiments, the data synchronization method may include the following steps.

[0106] S201, Get metadata.

[0107] refer to Figure 2 The metadata acquisition module 13 can be used to collect metadata. This embodiment of the disclosure provides two methods for collecting and maintaining metadata: automatic acquisition and manual entry. For example, metadata can be automatically acquired from the database metadata management module 11; or, for at least one of the source data sources of database type and system interface type, metadata can be obtained through manual entry.

[0108] In some embodiments, the acquired metadata is stored in a predetermined storage location of the metadata management subsystem, and the metadata query function is provided to the outside world through the metadata query interface.

[0109] S202, Establish the correspondence between the attribute information of the source data source and the target data source.

[0110] refer to Figure 2 The mapping configuration module 21 is used to obtain the correspondence between attribute information of the source data source and the target data source. In some embodiments, the mapping configuration function is implemented through the synchronization platform subsystem 20 to create the correspondence (also called mapping relationship) between attribute information of the source data source and the target data source. The mapping configuration function can be configured according to the following configuration steps.

[0111] Specifically, the configuration steps may include: selecting the data source to be configured, calling the metadata query interface, querying the data source to be mapped by the system interface name, for example, in this embodiment, the data source is a system interface, such as "Get User Information"; the system interface returns response structure information, and displays all fields of the response structure returned by the system interface on the page; the page provides a field selection function, for example, by using checkboxes to select specific fields to be synchronized, and all fields of the response structure are selected by default; the page responds to the page selection operation, obtains the selected field information as the attribute information of the source data source, which can also be called the data source field; selecting the target ES data source, and selecting the index of the ES data source, calling the metadata query interface to query the attribute list under the index, and displaying the attribute information in the attribute list under the index on the page, which can also be called the target field. In some embodiments, the data source field and the target field list are consistent to ensure a one-to-one correspondence between the fields.

[0112] S203, Create a synchronization configuration file.

[0113] refer to Figure 2 The mapping file creation module 22 is used to generate a synchronization configuration file based on the correspondence between the attribute information of the source data source and the target data source and the synchronization configuration parameters corresponding to the synchronization tool type.

[0114] In some embodiments, after determining the specific fields to be synchronized, the type of synchronization tool is selected, such as DataX or FlinkX. After selecting the synchronization tool, a synchronization configuration file (also known as a mapping file) corresponding to the tool type can be created. Simultaneously, configurations corresponding to different tool types can be configured. For example, if the synchronization tool is DataX, DataX-specific configurations could include rate limiting parameters, global concurrency parameters, etc. In some embodiments, the created configuration file can be in the form of a script file; the script file of the synchronization configuration file can be persisted to the device for storage and management to facilitate script migration.

[0115] S204, Deploy synchronized configuration files.

[0116] refer to Figure 2 The file deployment module 23 is used to deploy the synchronization configuration file to the synchronization tool to the specified file path; wherein, the synchronization tool has access permissions to the specified file path.

[0117] In some embodiments, after generating the synchronization configuration file, it is necessary to deploy the synchronization configuration file to a directory accessible to the synchronization tool. In this embodiment, NFS can be used to share and mount the synchronization configuration file, so that multiple instances of the synchronization tool can share and access the configuration file. By starting multiple synchronization instances, the synchronization efficiency can be improved.

[0118] S205, Configure the scheduling task and task execution parameters.

[0119] refer to Figure 2 The task scheduling module 24 is used to configure and schedule tasks and configure sharding parameters.

[0120] In some embodiments, the scheduling task is implemented using the open-source distributed task scheduling platform XXL-Job, and the scheduling task is created through a configured cron scheduling expression. When configuring task execution parameters, as an example, if the open-source version of the synchronization tool (such as DataX) does not support distributed processing, the target data to be synchronized is sharded by configuring sharding parameters. Specifically, if the source data source is a system interface type (i.e., an interface-type data source), sharding can be performed using pagination; if the source data source is a database type (e.g., a relational data source), sharding can be performed using data row identifiers and data row creation times.

[0121] S206, Start the data synchronization task.

[0122] refer to Figure 2 The task startup module 25 is used to start the data synchronization task.

[0123] In some embodiments, data synchronization tasks can be scheduled according to the configured cron expression. When the scheduled data synchronization task starts, the execution command is distributed to the proxy server according to the tool type of the selected synchronization tool. After receiving the execution command, the proxy server can start the local synchronization tool according to the configuration file, obtain the target data to be synchronized from the metadata, and write the obtained target data to the target data source.

[0124] S207, view the synchronization log and view the synchronization results.

[0125] refer to Figure 2 The viewing module 26 is used to view the synchronization logs and the synchronization results.

[0126] Specifically, the proxy server listens to log files and uploads the monitored log content to the display page provided by the metadata management subsystem 10 in real time. The logs are displayed and the results of the data synchronization task are viewed through the display page.

[0127] The data synchronization method of this disclosure can be adapted to commonly used different types of synchronization tools, thereby helping to shield the differences between different types of synchronization tools. When configuring synchronization tasks, synchronization configuration scripts for different types of synchronization tools can be generated based on the correspondence between the attribute information of the source data source and the target data source and the synchronization configuration parameters corresponding to the type of synchronization tool. When configuring and scheduling tasks, it can support the fragmentation of the target data to be synchronized, thereby improving the processing performance of data synchronization using synchronization tools.

[0128] During the data collection task execution, the task output logs and running results can be viewed, and any collection anomalies can be manually determined. If an anomaly is found, the data collection script or configuration file can be repaired, and the task can be restarted. If no anomalies are found, the data synchronization is complete. Thus, the system can automatically identify and handle synchronization anomalies. The script files of the synchronization configuration file can be persistently saved, and the historical information of data synchronization can be managed through the synchronization logs. In this embodiment, the lifecycle of the synchronization task can be managed through a page, such as task creation, configuring and scheduling tasks, configuring sharding parameters, viewing synchronization logs, and viewing synchronization results, thereby achieving multi-faceted management of the lifecycle of the synchronization task and improving the management efficiency of the lifecycle information of the synchronization task.

[0129] In addition, this disclosure also provides a data synchronization device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the data synchronization methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0130] Based on the above-described data synchronization method and data synchronization system embodiments, this application provides a data synchronization device, see [link to relevant documentation]. Figure 3 This is a block diagram of a data synchronization device provided in an embodiment of this disclosure. (Refer to...) Figure 3 This disclosure provides a data synchronization device 300, which includes:

[0131] The query module 310 is used to query metadata from the source data source through the metadata query interface and obtain the target data that needs to be synchronized from the metadata.

[0132] Configuration module 320 is used to generate and deploy a synchronization configuration file corresponding to the selected synchronization tool type based on the tool type of the selected synchronization tool.

[0133] The synchronization module 330 is used to synchronize the target data to the target data source based on the synchronization configuration file using the synchronization tool.

[0134] In some embodiments, metadata includes a data source name, entity information, and corresponding entity attribute information; the query module 310 is specifically used to: call a metadata query interface, query a source data source using the data source name, and obtain multiple candidate entity information and multiple candidate entity attribute information from the found source data source; display multiple candidate entity information and multiple candidate entity attribute information on a page; wherein the page contains multiple first control elements for marking multiple candidate entity information and multiple second control elements for marking multiple candidate entity attribute information; one candidate entity information corresponds to one first control element, and one candidate entity attribute information corresponds to one second control element; in response to operation instructions for multiple first control elements and multiple second control elements, the candidate entity information marked by the operated first control element and the candidate entity attribute information corresponding to the operated second control element are used as target data to be synchronized.

[0135] In some embodiments, the source data source type includes database type and system interface type; if the source data source type is database type, then the data source name is the database name, the entity information is the table name, and the entity attribute information is the field name in the corresponding table; if the source data source type is system interface type, then the data source name is the interface name, the entity information is the request structure information and response structure information of the system interface, and the entity attribute information is the field information in the corresponding request structure and the field information in the corresponding response structure.

[0136] In some embodiments, the metadata also includes: relevant attribute information corresponding to the data source type; when the source data source type is a database type, the relevant attribute information includes at least one of the following information items: database connection information and database login information; when the source data source type is a system interface type, the relevant attribute information includes at least one of the following information items: interface connection information and interface configuration information.

[0137] In some embodiments, the configuration module 320 is specifically used to: obtain the selected index information of the target data source, and call the metadata query interface to query the entity attribute information corresponding to the index information from the metadata using the index information; determine the one-to-one correspondence between the entity attribute information in the metadata and the entity attribute information corresponding to the index information; obtain the synchronization configuration parameters corresponding to the tool type according to the tool type of the synchronization tool; and generate a synchronization configuration file corresponding to the tool type according to the one-to-one correspondence and the synchronization configuration parameters, so as to record the synchronization correspondence between the entity attribute information in the target data and the entity attribute information corresponding to the index information through the synchronization configuration file.

[0138] In some embodiments, the data synchronization device further includes a file deployment module, specifically configured to: deploy a synchronization configuration file to a specified file path; wherein the synchronization tool has access permissions to the specified file path; wherein the deployment module is specifically configured to: mount the synchronization configuration file to a local file directory using a shared file method, so as to enable shared access to the synchronization configuration file locally.

[0139] In some embodiments, the synchronization module 330 is specifically used to: launch multiple synchronization instances of a pre-created synchronization tool; wherein each synchronization instance is used to synchronize target data; access the synchronization configuration file locally according to the multiple synchronization instances, and synchronize the target data to the target data source through the multiple synchronization instances jointly based on the synchronization configuration file.

[0140] In some embodiments, the synchronization module 330 is specifically used for: generating a synchronization task based on the received task scheduling expression; configuring task scheduling information based on the type of the source data source and the tool type; scheduling the synchronization task based on the task scheduling information; and synchronizing the target data to the target data source through the synchronization tool according to the synchronization configuration file when the scheduled synchronization task is in the started state.

[0141] In some embodiments, when configuring task scheduling information according to the type of the source data source and the tool type, the synchronization module 330 is specifically used to: determine the task execution mode supported by the synchronization tool according to the tool type, and set the task execution parameters corresponding to the task execution mode according to the type of the source data source; and generate task scheduling information corresponding to the task execution parameters.

[0142] In some embodiments, the task execution method includes a data sharding-based task execution method; when the synchronization module 330 sets the task execution parameters corresponding to the task execution method according to the type of the source data source, it is specifically used to: when the source data source type is a database type, use the paging parameter as the task execution parameter of the data sharding-based task execution method; wherein, the paging parameter is used to indicate that the target data is sharded by paging; when the source data source type is a system interface type, use the data row identifier and the data row creation time as the task execution parameters of the data sharding-based task execution method; wherein, the data row identifier and the data row creation time are used to indicate that the target data is sharded by the data row identifier and the data row creation time.

[0143] In some embodiments, the synchronization configuration file is used to record the synchronization correspondence between entity attribute information in the target data and entity attribute information in the target data source; when the synchronization module 330 is used to synchronize the target data to the target data source by means of the synchronization tool according to the synchronization configuration file when the scheduled synchronization task is in the started state, it is specifically used to: generate and send a synchronization command to the proxy server according to the type of the synchronization tool when the scheduled synchronization task is in the started state; wherein, the proxy server is used to start the synchronization tool according to the synchronization command, so as to obtain the synchronization correspondence through the synchronization tool, and execute the task scheduling information to write the target data to the target data source according to the synchronization correspondence.

[0144] In some embodiments, the data synchronization device 300 further includes: a log collection module, used to collect logs of the synchronization process through a proxy server during the process of synchronizing target data to the target data source based on the synchronization configuration file; and a log viewing module, used to send the collected log content to the data interface of the synchronization tool so that the log content can be displayed on a page through the synchronization tool.

[0145] According to the data synchronization apparatus of this disclosure, after obtaining metadata from the source data source through a metadata query interface, the target data to be synchronized can be selected from the metadata. Furthermore, a corresponding synchronization configuration file can be generated and deployed based on the type of the selected synchronization tool. The target data can then be synchronized from the source data source to the target data source using the synchronization tool and the synchronization configuration file. This data synchronization method can automatically generate synchronization configuration files for different types of synchronization tools through metadata, thus adapting to different types of synchronization tools. Moreover, it eliminates the need to develop and test data acquisition scripts, thereby reducing the workload of data synchronization and improving data synchronization efficiency.

[0146] Based on the above examples of data synchronization methods and embodiments of data synchronization devices, this disclosure provides an electronic device. See also... Figure 4 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. The electronic device includes: at least one processor 401; at least one memory 402; and one or more I / O interfaces 403 connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs that can be executed by the at least one processor 401, and the one or more computer programs are executed by the at least one processor 401 to enable the at least one processor 401 to perform the above-described data synchronization method.

[0147] This disclosure also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the aforementioned data synchronization method. The computer-readable storage medium may be volatile or non-volatile.

[0148] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described data synchronization method.

[0149] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0150] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0151] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0152] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0153] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0154] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0155] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0156] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0158] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A data synchronization method, characterized in that, include: Metadata is obtained from the source data source through the metadata query interface, and the target data to be synchronized is obtained from the metadata. Based on the tool type of the selected synchronization tool, obtain the synchronization configuration parameters corresponding to the tool type; Based on the one-to-one correspondence between the entity attribute information in the source data source and the entity attribute information in the target data source, and the synchronization configuration parameters, a synchronization configuration file corresponding to the tool type is generated and deployed. Different tool types correspond to different synchronization configuration files. The synchronization tool synchronizes the target data to the target data source based on the synchronization configuration file.

2. The method according to claim 1, characterized in that, The metadata includes the data source name, entity information, and corresponding entity attribute information; The process of obtaining metadata from the source data source through a metadata query interface and retrieving the target data to be synchronized from the metadata includes: Call the metadata query interface, use the data source name to query the source data source, and obtain the corresponding multiple candidate entity information and multiple candidate entity attribute information from the found source data source; The page displays the multiple candidate entity information and the multiple candidate entity attribute information; wherein, the page includes multiple first control elements for marking the multiple candidate entity information and multiple second control elements for marking the multiple candidate entity attribute information; one candidate entity information corresponds to one first control element, and one candidate entity attribute information corresponds to one second control element; In response to operation instructions targeting multiple first control elements and multiple second control elements, the candidate entity information marked by the operated first control element and the candidate entity attribute information corresponding to the operated second control element are used as target data to be synchronized.

3. The method according to claim 2, characterized in that, The types of the source data include database types and system interface types; If the source data type is a database type, then the data source name is the database name, the entity information is the table name, and the entity attribute information is the field name in the corresponding table; If the source data type is a system interface type, then the data source name is the interface name, the entity information is the request structure information and response structure information of the system interface, and the entity attribute information is the field information in the corresponding request structure and the field information in the corresponding response structure.

4. The method according to claim 3, characterized in that, The metadata also includes: relevant attribute information corresponding to the data source type; When the source data type is a database, the relevant attribute information includes at least one of the following items: database connection information and database login information; When the source data type is a system interface type, the relevant attribute information includes at least one of the following information items: interface link information and interface configuration information.

5. The method according to claim 1, characterized in that, The step of generating and deploying a synchronization configuration file corresponding to the tool type based on the one-to-one correspondence between entity attribute information in the source data source and entity attribute information in the target data source, and the synchronization configuration parameters, includes: Obtain the selected index information of the target data source, and call the metadata query interface to use the index information to query the entity attribute information corresponding to the index information from the metadata; Determine the one-to-one correspondence between the entity attribute information in the metadata and the entity attribute information corresponding to the index information; Based on the one-to-one correspondence and the synchronization configuration parameters, a synchronization configuration file corresponding to the tool type is generated to record the synchronization correspondence between the entity attribute information in the target data and the entity attribute information corresponding to the index information.

6. The method according to claim 5, characterized in that, The method further includes: The synchronization configuration file is deployed to a specified file path; wherein the synchronization tool has access permissions to the specified file path; The step of deploying the synchronization configuration file to a specified file path includes: mounting the synchronization configuration file to a local file directory using a shared file method, so as to enable local shared access to the synchronization configuration file; The step of synchronizing the target data to the target data source using the synchronization tool based on the synchronization configuration file includes: Launch multiple synchronization instances of the pre-created synchronization tool; each synchronization instance is used to synchronize the target data. The multiple synchronization instances share access to the synchronization configuration file locally, and the target data is synchronized to the target data source by the multiple synchronization instances working together based on the synchronization configuration file.

7. The method according to claim 1, characterized in that, The step of synchronizing the target data to the target data source using the synchronization tool based on the synchronization configuration file includes: Generate a synchronization task based on the received task scheduling expression; Configure task scheduling information according to the type of the source data source and the type of the tool; The synchronization task is scheduled based on the task scheduling information; When the scheduled synchronization task is in the started state, the synchronization tool synchronizes the target data to the target data source according to the synchronization configuration file.

8. The method according to claim 7, characterized in that, The step of configuring task scheduling information based on the type of the source data source and the type of the tool includes: The task execution methods supported by the synchronization tool are determined according to the tool type, and the task execution parameters corresponding to the task execution methods are set according to the source data type. Generate task scheduling information corresponding to the task execution parameters.

9. The method according to claim 8, characterized in that, The task execution method includes a data sharding-based task execution method; setting the task execution parameters corresponding to the task execution method according to the type of the source data source includes: When the source data type is a database, the pagination parameter is used as the task execution parameter for the data sharding-based task execution method; wherein, the pagination parameter is used to indicate that the target data is sharded by pagination. When the source data type is a system interface type, the data row identifier and data row creation time are used as task execution parameters for the data sharding-based task execution method; wherein, the data row identifier and the data row creation time are used to indicate that the target data is sharded according to the data row identifier and the data row creation time.

10. The method according to claim 7, characterized in that, The synchronization configuration file is used to record the synchronization correspondence between the entity attribute information in the target data and the entity attribute information in the target data source; When the scheduled synchronization task is in the started state, the synchronization tool synchronizes the target data to the target data source according to the synchronization configuration file, including: When the scheduled synchronization task is in the started state, a synchronization command is generated and sent to the proxy server according to the type of the synchronization tool; The proxy server is used to start the synchronization tool according to the synchronization command, so as to obtain the synchronization correspondence through the synchronization tool and execute the task scheduling information to write the target data into the target data source according to the synchronization correspondence.

11. The method according to claim 1, characterized in that, The method further includes: During the synchronization process of synchronizing the target data to the target data source based on the synchronization configuration file, logs of the synchronization process are collected through a proxy server; The collected log content is sent to the data interface of the synchronization tool so that the log content can be displayed on a page through the synchronization tool.

12. A data synchronization device, characterized in that, include: The query module is used to query metadata from the source data source through the metadata query interface, and to obtain the target data that needs to be synchronized from the metadata. The configuration module is used to obtain the synchronization configuration parameters corresponding to the selected synchronization tool type based on the tool type of the selected synchronization tool; and to generate and deploy the synchronization configuration file corresponding to the tool type based on the one-to-one correspondence between the entity attribute information in the source data source and the entity attribute information in the target data source and the synchronization configuration parameters. Different tool types correspond to different synchronization configuration files. The synchronization module is used to synchronize the target data to the target data source based on the synchronization configuration file using the synchronization tool.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the data synchronization method as described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the data synchronization method as described in any one of claims 1-11.

Images