Data transmission method and device among multiple data sources and electronic equipment

Abstract

The invention discloses a data transmission method and device among multiple data sources and electronic equipment, and relates to the technical field of big data or other related fields, the method comprises the following steps: receiving data source information of a to-be-synchronized data source and a target data source, and establishing communication connection between the to-be-synchronized data source and the target data source; receiving a transmission mode and a storage path, and determining target transmission data based on the transmission mode, the target transmission data comprising one or more target transmission files; for each target transfer file, exporting the target transfer file from the to-be-synchronized data source to the storage path in batches, and importing the target transfer file to the target data source when it is detected that all the target transfer files are exported; and when it is detected that all the target transmission data are imported into the target data source, performing data verification on the target transmission data in the target data source, and outputting a verification result. According to the invention, the technical problems of poor transmission stability and low efficiency caused by the fact that data transmission depends on overall transmission in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology or other related fields, and in particular to a data transmission method and device, and an electronic device between multiple data sources. Background Art

[0002] In the current context of digital transformation, data has become a key asset to drive corporate decision-making, optimize business processes, and improve service efficiency. With the growing demand for data integration and migration, especially in industries such as finance, telecommunications, and the Internet, data transmission across platforms and systems has become more and more frequent. This demand has led to the rapid development of data transmission tools, which aim to achieve data transmission between different data sources to support business continuity and unified data management.

[0003] In the related art, the existing data transmission strategy between multiple data sources is to first export the target data from one data source and put it into a specified directory, and then import the target data from the instruction directory to the target data source. However, the existing data transmission strategy between multiple data sources has many disadvantages. For example, the cost is high. The existing data transmission strategy between multiple data sources requires a lot of preliminary preparation, including space estimation of the synchronization data source and the target data source, export directory space estimation, export file size estimation, etc. This not only consumes a lot of manpower and time resources, but also may increase additional storage and network costs due to inaccurate evaluation. Secondly, the efficiency is low. Especially when processing large-scale data, exporting and importing large files at one time is not only inefficient, but also slows down the data transmission speed and occupies too many resources due to bottlenecks in database performance. Finally, the stability is insufficient. Any interruption in the data transmission process, such as network failure, data format mismatch or insufficient storage space, may lead to data migration failure, and then the data needs to be transmitted from the beginning, which greatly reduces the efficiency of data transmission.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present invention provide a method and device for data transmission between multiple data sources, and an electronic device, so as to at least solve the technical problem in the related art that data transmission depends on overall transmission, resulting in poor transmission stability and low efficiency.

[0006] In order to achieve the above-mentioned purpose, according to one aspect of the present application, a method for data transmission between multiple data sources is provided, which is applied to a pre-built data transmission tool, including: receiving data source information of a data source to be synchronized and a target data source input by an external terminal, and establishing a communication connection between the data source to be synchronized and the target data source, wherein the data source information includes: server address, port, database type, database name, database account and password; receiving a transmission mode and storage path selected by the external terminal, and determining target transmission data based on the transmission mode, wherein the transmission mode includes: transmission by table, transmission by user and transmission by database, and the target transmission data includes one or more target transmission files; for each target transmission file, exporting the target transmission file from the data source to be synchronized to the storage path in batches, and when it is detected that all the target transmission files are exported, importing the target transmission file from the storage path to the target data source; when it is detected that all the target transmission data are imported into the target data source, data verification is performed on the target transmission data in the target data source, and the verification result is output.

[0007] Optionally, the step of establishing a communication connection between the data source to be synchronized and the target data source includes: obtaining a configuration file of the data source to be synchronized, and changing the configuration file of the data source to be synchronized based on data source information of the data source to be synchronized and the target data source; obtaining a configuration file of the target data source, and changing the configuration file of the target data source based on data source information of the data source to be synchronized and the target data source; establishing a communication connection between the data source to be synchronized and the target data source based on the configuration files of the data source to be synchronized and the target data source, and outputting the connection result.

[0008] Optionally, the step of determining the target transmission data based on the transmission mode includes: when the transmission mode is transmission by table, querying all tables under the data source to be synchronized, and outputting all the queried tables, receiving a table selected from all the tables by an external terminal, and determining the selected table as the target transmission data; when the transmission mode is transmission by user, querying all users under the data source to be synchronized, and outputting all the queried users, and then receiving a user selected from all the users by the external terminal, and determining all tables under the selected user as the target transmission data; when the transmission mode is transmission by database, all tables under the data source to be synchronized are determined as the target transmission data.

[0009] Optionally, before exporting the target transfer file from the data source to be synchronized, it also includes: determining whether there is a target user and a target table under the target user in the target data source; if the target user does not exist in the target data source, creating a new target user and a target table under the target user in the target data source; if the target user exists in the target data source but there is no target table under the target user, creating a new target table under the target user in the target data source.

[0010] Optionally, the step of importing the target transfer file from the storage path to the target data source includes: detecting whether the target transfer file has been completely exported to the storage path to obtain a first detection result; when the first detection result indicates that the target transfer file has been completely exported, calling a pre-built import program to import the target transfer file from the storage path to the target table of the target data source.

[0011] Optionally, after all the target transfer files are imported into the target data source, the method includes: detecting whether all the target transfer files have been imported into the target data source to obtain a second detection result; when the second detection result indicates that all the target transfer files have been imported into the target data source, deleting the target transfer files in the data source to be synchronized and the storage path.

[0012] Optionally, the step of performing data verification on the target transmission data in the target data source includes: comparing the data volume of the target transmission data in the data source to be synchronized with that in the target data source, and when the comparison result indicates that the data volume is consistent, determining that the data verification has passed; or, comparing the field data of each table in the target transmission data in the data source to be synchronized with that in the target data source, and when the comparison result indicates that the field data of all tables are consistent, determining that the data verification has passed.

[0013] According to another aspect of an embodiment of the present invention, there is also provided a data transmission device between multiple data sources, which is applied to a pre-built data transmission tool, including: a connection unit, which is used to receive data source information of a data source to be synchronized and a target data source input by an external terminal, and establish a communication connection between the data source to be synchronized and the target data source, wherein the data source information includes: a server address, a port, a database type, a database name, a database account and a password; a determination unit, which is used to receive a transmission mode and a storage path selected by an external terminal, and determine target transmission data based on the transmission mode, wherein the transmission mode includes: transmission by table, transmission by user and transmission by database, and the target transmission data includes one or more target transmission files; a transmission unit, which, for each target transmission file, exports the target transmission file from the data source to be synchronized to the storage path in batches, and when it is detected that all the target transmission files are exported, imports the target transmission file from the storage path to the target data source; a verification unit, which is used to perform data verification on the target transmission data in the target data source when it is detected that all the target transmission data are imported into the target data source, and outputs a verification result.

[0014] Optionally, the connection unit includes: a first change module, used to obtain the configuration file of the data source to be synchronized, and change the configuration file of the data source to be synchronized based on the data source information of the data source to be synchronized and the target data source; a second change module, used to obtain the configuration file of the target data source, and change the configuration file of the target data source based on the data source information of the data source to be synchronized and the target data source; an establishment module, which establishes a communication connection between the data source to be synchronized and the target data source based on the configuration files of the data source to be synchronized and the target data source, and outputs the connection result.

[0015] Optionally, the determination unit includes: a first determination module, which is used to query all tables under the data source to be synchronized when the transmission mode is transmission by table, and output all the queried tables, receive a table selected by an external terminal from all the tables, and determine the selected table as the target transmission data; a second determination module, which is used to query all users under the data source to be synchronized when the transmission mode is transmission by user, and output all the queried users, then receive a user selected by the external terminal from all the users, and determine all tables under the selected user as the target transmission data; a third determination module, which is used to determine all tables under the data source to be synchronized as the target transmission data when the transmission mode is transmission by database.

[0016] Optionally, the transmission unit includes: a judgment module, used to judge whether a target user and a target table under the target user exist in the target data source; a first creation module, used to create a target user and a target table under the target user in the target data source if the target user does not exist in the target data source; and a second creation module, used to create a target table under the target user in the target data source if the target user exists in the target data source but the target table under the target user does not exist.

[0017] Optionally, the transmission unit also includes: a first detection module, used to detect whether the target transmission files have all been exported to the storage path to obtain a first detection result; an import module, used to call a pre-built import program to import the target transmission files from the storage path to the target table of the target data source when the first detection result indicates that the target transmission files have all been exported.

[0018] Optionally, the transmission unit also includes: a second detection module, used to detect whether the target transmission files have been completely imported into the target data source to obtain a second detection result; a deletion module, used to delete the target transmission files in the data source to be synchronized and the storage path when the second detection result indicates that the target transmission files have been completely imported into the target data source.

[0019] Optionally, the verification unit includes: a first verification module, used to compare the data volume of the target transmission data in the data source to be synchronized and the target data source, and when the comparison result indicates that the data volume is consistent, it is determined that the data verification has passed; a second verification module, used to compare the field data of each table in the target transmission data in the data source to be synchronized and the target data source, and when the comparison result indicates that the field data of all tables are consistent, it is determined that the data verification has passed.

[0020] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned methods for transmitting data between multiple data sources.

[0021] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any of the above-mentioned methods for transmitting data between multiple data sources.

[0022] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program, which implements the steps of any one of the above-mentioned methods for transmitting data between multiple data sources when executed by a processor.

[0023] In the present disclosure, a data transmission method between multiple data sources is applied to a pre-built data transmission tool, which provides a friendly graphical interface for external terminals or operation objects, allowing the external terminals or operation objects to select data sources, specify transmission methods, and provide practical functions such as test connection, transmission progress monitoring, and data verification. By providing an intuitive and highly interactive operation interface, the learning cost of external terminals or operation objects is reduced, greatly improving the user experience of the data transmission tool and the popularity of the data transmission tool.

[0024] In the present disclosure, by receiving data source information of a data source to be synchronized and a target data source input by an external terminal, a communication connection between the data source to be synchronized and the target data source is established, and then by receiving a transmission mode and a storage path selected by the external terminal, the target transmission data is determined based on the transmission mode, wherein the target transmission data includes one or more target transmission files, and for each target transmission file, the target transmission file is exported from the data source to be synchronized to the storage path in batches, and when it is detected that all the target transmission files are exported, the target transmission files are imported from the storage path to the target data source, and finally, when it is detected that all the target transmission data are imported into the target data source, data verification is performed on the target transmission data in the target data source, and the verification result is output, thereby completing the transmission process of the target transmission data from the data source to be synchronized to the target data source.

[0025] According to the above disclosed content, the target transmission data includes one or more target transmission files, each target transmission file is exported and imported in batches, and the export and import process of each target transmission file is performed independently, that is, multiple target transmission files can be exported and imported at the same time. This transmission method significantly improves the transmission efficiency of the target transmission data. At the same time, the tables and fields in the target data source are updated based on the tables and fields in the data source to be synchronized, avoiding data update failures due to differences between the tables or fields of the target data source and the data source to be synchronized, significantly reducing the error probability of data transmission, and thus solving the technical problem in the related technology that data transmission depends on overall transmission, resulting in poor transmission stability and low efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a data transmission method between multiple data sources is shown;

[0028] Figure 2 is a flowchart of an optional method for data transmission between multiple data sources according to an embodiment of the present invention;

[0029] Figure 3 is a schematic diagram of a connection interface of an optional data transmission tool between multiple data sources according to an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of an interface for selecting a transmission mode of an optional data transmission tool between multiple data sources according to an embodiment of the present invention;

[0031] Figure 5 is a schematic diagram of a transmission interface of an optional data transmission tool between multiple data sources according to an embodiment of the present invention;

[0032] Figure 6 is a schematic diagram of a verification interface of an optional data transmission tool between multiple data sources according to an embodiment of the present invention;

[0033] Figure 7 is a schematic diagram of an optional data transmission device between multiple data sources according to an embodiment of the present invention;

[0034] Figure 8 It is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] It should be noted that the data transmission method and apparatus between multiple data sources in the present disclosure can be used in the field of big data technology when data transmission is performed between multiple data sources, and can also be used in any field other than the field of big data technology when data transmission is performed between multiple data sources. The present disclosure does not limit the application field of the data transmission method and apparatus between multiple data sources.

[0038] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0039] It should be noted that in the present disclosure, when collecting and analyzing customer information, corresponding operation entrances are provided for users to choose to agree or reject the automated decision results; if the user chooses to reject, the expert decision-making process is entered.

[0040] The following embodiments of the present invention can be applied to systems / applications / devices for data transmission between various multiple data sources. In the present invention, the target transmission data includes one or more target transmission files, each target transmission file is exported and imported in batches, and the export and import process of each target transmission file is performed independently, that is, multiple target transmission files can be exported and imported at the same time. This transmission method significantly improves the transmission efficiency of the target transmission data. At the same time, the tables and fields in the target data source are updated based on the tables and fields in the data source to be synchronized, avoiding data update failures due to differences in tables or fields between the target data source and the data source to be synchronized, and significantly reducing the error probability of data transmission.

[0041] The present invention is described in detail below in conjunction with various embodiments.

[0042] Embodiment 1

[0043] According to an embodiment of the present invention, an embodiment of a method for transmitting data between multiple data sources is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] The data transmission method embodiment between multiple data sources provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a data transmission method between multiple data sources is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more ( Figure 1 The computer 102 includes a processor 102 (the processor 102 may include but is not limited to a microprocessor MCU (Microcontroller Unit) or a programmable logic device FPGA (Field Programmable Gate Array)), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS), a network interface, a power supply and / or a camera. Those skilled in the art will understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0045] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0046] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the data transmission method between multiple data sources in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, to implement the above-mentioned data transmission method between multiple data sources. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0048] The display may be, for example, a touch screen liquid crystal display (LCD), which may enable a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0049] Under the above operating environment, this application provides Figure 2 The data transmission method between multiple data sources is shown. Figure 2 is a flowchart of an optional data transmission method between multiple data sources according to an embodiment of the present invention. Figure 2 As shown, the method comprises the following steps:

[0050] Step S201, receiving data source information of the data source to be synchronized and the target data source input by the external terminal, and establishing a communication connection between the data source to be synchronized and the target data source, wherein the data source information includes: server address, port, database type, database name, database account and password.

[0051] First of all, it should be noted that the method provided in the embodiment of the present invention is applied to a pre-built data transmission tool. The data transmission tool is configured with jar packages required for connecting to various relevant data sources, including connection jar packages for various types of databases. At the same time, the data transmission tool provides an intuitive operation platform, and an external terminal or operation object can input or select relevant data on the front-end interface of the operation platform.

[0052] In an embodiment of the present invention, it is first necessary to determine the data source to be synchronized and the target data source, and establish a communication connection between the two data sources. When establishing the connection, it is necessary to obtain the data source information of the data source to be synchronized and the target data source, including the server address, port, database type, database name, database account and password.

[0053] Figure 3 is a schematic diagram of a connection interface of an optional data transmission tool between multiple data sources according to an embodiment of the present invention, such as Figure 3As shown, the external terminal or operation object selects the database type of the data source to be synchronized and the target data source through the "(Drop-down selection)" box, and then enters the server address, port, database account, database password and database name of the data source to be synchronized and the target data source through the "Server address" input box, "Port" input box, "User name" input box, "Password" input box and "Database" input box respectively.

[0054] Next, click on the external terminal or operation object Figure 3 After clicking the "Test Connection" button shown, the embodiment of the present invention uses the received server address, port, database type, database name, database account and password to try to establish a communication connection between the data source to be synchronized and the target data source. During this process, the embodiment of the present invention will perform a connection test to verify whether the provided information is accurate and whether the established communication connection is stable and reliable. If the connection fails, Figure 3 The "Test Result" output box shown below outputs the failure information so that the external terminal or operation object can correct the error information in time and try to establish the connection again. If the connection is successful, the "Test Result" output box outputs the success information. After the connection is successful, the external terminal or operation object can click Figure 3 Click the "Transfer" button shown to enter the data transmission method selection stage.

[0055] The embodiment of the present invention effectively establishes a communication bridge between the data source to be synchronized and the target data source, providing a solid foundation for subsequent data migration and verification. The process of establishing the connection of the data source is automated and intelligent, which not only simplifies the operation of the external terminal or operation object, but also significantly improves the efficiency and stability of data transmission.

[0056] Step S202, receiving the transmission mode and storage path selected by the external terminal, and determining the target transmission data based on the transmission mode, wherein the transmission mode includes: transmission by table, transmission by user and transmission by database, and the target transmission data includes one or more target transmission files.

[0057] In the data transmission mode selection stage, the embodiment of the present invention first needs to receive the transmission mode and storage path selected by the external terminal or the operation object. Figure 4 is a schematic diagram of an interface for selecting a transmission mode of a data transmission tool between multiple data sources according to an embodiment of the present invention, wherein an external terminal or an operation object is Figure 4 Select the data transmission method in the "Transfer method (drop-down selection)" box shown.

[0058] The transmission modes include: transmission by table, transmission by user, and transmission by database. When the transmission mode by table is selected, the external terminal or operation object selects a specific table name. The embodiment of the present invention will only perform data migration for the selected table. The transmission mode by table is suitable for updating or migrating individual tables, reducing unnecessary data processing and improving pertinence and efficiency. When the transmission mode by user is selected, the external terminal or operation object selects one or more specific data source users. The embodiment of the present invention will migrate the data of the user and all tables under it. The transmission mode by user is particularly suitable for user-level data migration or update, and can ensure the integrity and consistency of user data. When the transmission mode by database is selected, the external terminal or operation object chooses to migrate the data of the entire database. The embodiment of the present invention will automatically process the data migration of all users in the data source and all tables under it. For scenarios that require full database migration, the transmission mode by database provides the most comprehensive and convenient solution strategy.

[0059] Based on the selected transmission mode, the embodiment of the present invention automatically determines the target transmission data, that is, the database content that needs to be migrated. The embodiment of the present invention determines the target transmission data that needs to be migrated through the table name or user selected by the external terminal or the operation object. It should be noted that the target transmission data usually includes one or more target transmission files, that is, the data is transmitted in the form of files.

[0060] At the same time, the external terminal or operation object also needs to specify a storage path for temporary storage of exported data files. Figure 4 In the "Path (Please Select)" box shown, select the storage path in the local folder that pops up.

[0061] After determining the transmission method and target transmission data, the external terminal or operation object clicks Figure 4 Click "Please confirm" to enter the data transmission stage.

[0062] It should be noted that the external terminal or operation object is clicked Figure 4 Click the "Back" button to return to the connection interface and modify the connection information.

[0063] Step S203, for each target transmission file, the target transmission file is exported from the to-be-synchronized data source to the storage path in batches, and when it is detected that all the target transmission files are exported, the target transmission files are imported from the storage path to the target data source.

[0064] In the data transmission stage, the embodiment of the present invention adopts a data batch processing and dynamic monitoring strategy. For each target transmission file in the target transmission data, the embodiment of the present invention first divides the target transmission file into multiple smaller data batches. For example, if a target transmission file has a data volume of 10,000, it can be divided into 5 parts, each with a data volume of 2,000.

[0065] Subsequently, the embodiment of the present invention exports each batch of data from the data source to be synchronized to the storage path specified by the external terminal or the operation object. This storage path serves as a temporary storage area for data transmission, ensuring the safe storage of data files and subsequent efficient reading. During the export process, the embodiment of the present invention continuously monitors the progress and status of the data export. Figure 5 is a schematic diagram of a transmission interface of an optional data transmission tool between multiple data sources according to an embodiment of the present invention. In this embodiment of the present invention, the exported information is displayed on Figure 5 In the "Export" box shown in the figure, the embodiment of the present invention adopts a loop mechanism, that is, for a target transmission file divided into five data portions, the export process will be executed five times in a loop until the entire target transmission file is exported to the storage space under the storage path.

[0066] When it is detected that all target transfer files have been exported to the storage path, the embodiment of the present invention will immediately start the data import process to import the target data files from the storage path to the target data source. This import process also follows the batch principle to ensure that the data can be quickly and smoothly imported into the target database, and the import information is displayed on the Figure 5 in the Import box that appears.

[0067] It should be noted that, in an embodiment of the present invention, the export and import process of each target data file is performed independently, and the import and export processes of multiple target data files can be performed synchronously. There is no need to wait for the entire target transmission data (including multiple target data files) to be exported and then imported into the target data source, which greatly improves the efficiency of data transmission.

[0068] It should also be noted that external terminals or operation objects can be opened by clicking Figure 5 Click the "Stop" button to terminate the export and import process. Figure 5 The "Back" button shown can return to the transmission method selection interface to re-determine the transmission method, target transmission data and storage path.

[0069] After data transfer is completed, the external terminal or operation object can be opened by clicking Figure 5 Click the "Verify" button shown to enter the verification stage.

[0070] Step S204: when it is detected that all the target transmission data are imported into the target data source, data verification is performed on the target transmission data in the target data source, and a verification result is output.

[0071] In the verification phase, after detecting that all the target transmission data have been imported into the target data source, the embodiment of the present invention performs data verification on the target transmission data in the target data source. Figure 6 This is a schematic diagram of the verification interface of an optional data transmission tool between multiple data sources according to an embodiment of the present invention. There are two data verification methods, namely data verification and quality verification. The external terminal or the operation object starts data verification or quality verification by clicking the "Data Verification" button or the "Quality Verification" button to check the integrity and accuracy of data migration and ensure high efficiency and high accuracy of transmission between data sources. Finally, the verification results are displayed on Figure 6 The "Verification Results" output box is shown.

[0072] Optionally, the step of establishing a communication connection between the data source to be synchronized and the target data source includes: obtaining a configuration file of the data source to be synchronized, and changing the configuration file of the data source to be synchronized based on data source information of the data source to be synchronized and the target data source; obtaining a configuration file of the target data source, and changing the configuration file of the target data source based on data source information of the data source to be synchronized and the target data source; establishing a communication connection between the data source to be synchronized and the target data source based on the configuration files of the data source to be synchronized and the target data source, and outputting the connection result.

[0073] In step S201, when establishing a communication connection between the data source to be synchronized and the target data source, the configuration files of the data source to be synchronized and the target data source are first read from the internal storage of the data transmission tool. These configuration files contain the necessary parameters and standard connection information for communicating with various types of databases. Subsequently, the embodiment of the present invention automatically changes the relevant parameters in the configuration files of the data source to be synchronized and the target data source based on the received data source information of the data source to be synchronized and the target data source, including the server address, port number, database type, database name, database account and password, to ensure that a correct communication connection is established between the data source to be synchronized and the target data source. After the configuration file change is completed, a communication connection between the data source to be synchronized and the target data source is established based on the configuration files of the data source to be synchronized and the target data source, and at the same time, the connection result is displayed on Figure 3 The "Test Results" output box is shown.

[0074] Optionally, the step of determining the target transmission data based on the transmission mode includes: when the transmission mode is transmission by table, querying all tables under the data source to be synchronized, and outputting all the queried tables, receiving a table selected from all the tables by an external terminal, and determining the selected table as the target transmission data; when the transmission mode is transmission by user, querying all users under the data source to be synchronized, and outputting all the queried users, and then receiving a user selected from all the users by the external terminal, and determining all tables under the selected user as the target transmission data; when the transmission mode is transmission by database, all tables under the data source to be synchronized are determined as the target transmission data.

[0075] In step S202, when the target transmission data is determined based on the transmission mode, if the transmission mode is table-based transmission, the embodiment of the present invention will query all tables under the data source to be synchronized, and display all the queried tables in Figure 4 In the "Table Name (Drop-Down Selection)" menu shown in the figure, the external terminal or the operation object selects the table name to be transmitted in the menu, and the embodiment of the present invention determines the selected table as the target transmission data. After determining the target transmission data, the external terminal or the operation object clicks Figure 4 Click the "Please Confirm" button to enter the data transmission stage.

[0076] If the transmission mode is transmission by user, the embodiment of the present invention will query all users under the data source to be synchronized, and display all the queried users in Figure 4 In the "User (drop-down selection)" menu shown in the figure, the external terminal or the operation object selects one or more users in the menu, and the embodiment of the present invention determines all tables under the selected user as target transmission data. After determining the target transmission data, the external terminal or the operation object clicks Figure 4 Click the "Please Confirm" button to enter the data transmission stage.

[0077] If the transmission mode is to transmit by database, the embodiment of the present invention determines all tables under the data source to be synchronized as the target transmission data. Figure 4 Click the "Please Confirm" button to enter the data transmission stage.

[0078] Optionally, before exporting the target transfer file from the data source to be synchronized, it also includes: determining whether there is a target user and a target table under the target user in the target data source; if the target user does not exist in the target data source, creating a new target user and a target table under the target user in the target data source; if the target user exists in the target data source but there is no target table under the target user, creating a new target table under the target user in the target data source.

[0079] In step S203, before exporting the target transmission file from the data source to be synchronized, it is determined whether a corresponding target table exists in the target data source.

[0080] If the transmission is by table name, the embodiment of the present invention will query which user the selected table is under in the data source to be synchronized, and determine whether the same user exists in the target data source. If the same user does not exist, a new same user will be created and a table with the same table name will be created under this user. If the same user exists but a table with the same table name does not exist, a new table with the same table name will be created under this user. If the same user exists and a table with the same table name exists under this user, the target transmission data will be directly imported into this table.

[0081] If you choose to transfer by user or by database, the embodiment of the present invention will determine whether the same user and the same table exist in the target data source. If the same user does not exist, a new user will be created and all tables under this user will be created. If the same user exists but one or more of the same tables are missing, the missing tables will be created under the user. If the same user exists and all the same tables exist, the target transfer data will be directly imported into the user.

[0082] Before exporting the target transmission data from the data source to be synchronized, the embodiment of the present invention determines in advance whether the target data source has the same storage space, and creates the same storage space if the same storage space does not exist, thereby ensuring that the target data source is ready to receive the target transmission data and avoiding potential errors in the data transmission process.

[0083] Optionally, the step of importing the target transfer file from the storage path to the target data source includes: detecting whether the target transfer file has been completely exported to the storage path to obtain a first detection result; when the first detection result indicates that the target transfer file has been completely exported, calling a pre-built import program to import the target transfer file from the storage path to the target table of the target data source.

[0084] In step S203, when the target transmission file is imported from the storage path to the target data source, the embodiment of the present invention will continuously detect the progress of exporting the target transmission file from the data source to be synchronized to the specified storage path, determine whether all data batches have been successfully exported, and generate a first detection result. When the first detection result indicates that the target transmission file has been completely exported, the embodiment of the present invention will automatically call a pre-built import program to accurately and quickly import the data in the target transmission file into the target table of the target data source.

[0085] Optionally, after all the target transfer files are imported into the target data source, the method includes: detecting whether all the target transfer files have been imported into the target data source to obtain a second detection result; when the second detection result indicates that all the target transfer files have been imported into the target data source, deleting the target transfer files in the data source to be synchronized and the storage path.

[0086] In step S203, after all the target transmission files are imported into the target data source, the embodiment of the present invention will continue to detect the progress of the target transmission files being imported from the specified storage path to the target data source, determine whether all data batches have been successfully imported, and generate a second detection result. If the second detection result indicates that all the target transmission files have been imported into the target data source, the target transmission files in the data source to be synchronized and the storage path are deleted to ensure efficient use of storage space.

[0087] Optionally, the step of performing data verification on the target transmission data in the target data source includes: comparing the data volume of the target transmission data in the data source to be synchronized with that in the target data source, and when the comparison result indicates that the data volume is consistent, determining that the data verification has passed; or, comparing the field data of each table in the target transmission data in the data source to be synchronized with that in the target data source, and when the comparison result indicates that the field data of all tables are consistent, determining that the data verification has passed.

[0088] In step S204, the data verification process includes two main verification methods, namely data verification and quality verification. Data verification compares the data volume of the target data to be synchronized with the target data source to ensure that the data volume of the two is consistent, and then determines whether any data is omitted or repeated during the data migration process. This verification method is fast and effective, and can give a judgment on whether the data volume matches in a short time.

[0089] Quality verification will compare the field data of each table in the target data source to be synchronized with the target data source to check the consistency of the data content. This verification method can be detailed to the data of each field to ensure that the content of the migrated data is completely consistent with the data before migration.

[0090] During the data verification process, external terminals or operation objects can Figure 6 The "Verify table name: user" button, "Verify table name: table name" button or "Verify table name: database" button can be used to select verification by user, by table or by the entire database. The embodiment of the present invention will perform data verification or quality verification according to different selections. After the target transmission data is imported, the embodiment of the present invention automatically performs data verification or quality verification and outputs detailed verification results, providing a strong guarantee for the final quality of data transmission.

[0091] In the embodiment of the present invention, the data transmission method between multiple data sources is applied to a pre-built data transmission tool, which provides a friendly graphical interface for external terminals or operation objects, allowing the external terminals or operation objects to select data sources, specify transmission methods, and provide practical functions such as test connection, transmission progress monitoring, and data verification. By providing an intuitive and highly interactive operation interface, the learning cost of external terminals or operation objects is reduced, and the user experience and popularity of the data transmission tool are greatly improved.

[0092] The above-mentioned steps of the method for data transmission between multiple data sources are as follows: by receiving the data source information of the data source to be synchronized and the target data source input by the external terminal, a communication connection is established between the data source to be synchronized and the target data source; then, by receiving the transmission mode and storage path selected by the external terminal, the target transmission data is determined based on the transmission mode, wherein the target transmission data includes one or more target transmission files; for each target transmission file, the target transmission file is exported from the data source to be synchronized to the storage path in batches; when it is detected that all the target transmission files are exported, the target transmission files are imported from the storage path to the target data source; finally, when it is detected that all the target transmission data are imported into the target data source, data verification is performed on the target transmission data in the target data source, and the verification result is output, thereby completing the transmission process of the target transmission data from the data source to be synchronized to the target data source.

[0093] In the embodiment of the present invention, the tables and fields of the target data source are updated based on the tables and fields of the data source to be synchronized, which significantly reduces the amount of preparation work before data migration and avoids the tedious process of manually checking the table structure, field type and data volume, thereby reducing the data transmission cost.

[0094] In an embodiment of the present invention, the target transmission data includes one or more target transmission files, each target transmission file is exported and imported in batches, and the export and import process of each target transmission file is performed independently, that is, multiple target transmission files can be exported and imported at the same time. This transmission method significantly improves the transmission efficiency of the target transmission data. At the same time, the tables and fields in the target data source are updated based on the tables and fields in the data source to be synchronized, avoiding data update failures caused by differences between the tables or fields of the target data source and the data source to be synchronized, significantly reducing the error probability of data transmission, and thus solving the technical problem in the related technology that data transmission depends on overall transmission, resulting in poor transmission stability and low efficiency.

[0095] At the same time, in the embodiment of the present invention, multiple target transmission files are simultaneously exported and imported. After a single target transmission file is imported, the target transmission file in the data source to be synchronized and the storage path will be deleted in time, saving storage space and improving transmission efficiency. After the export and import process of the target transmission data is completed, the target transmission data is verified or quality verified, and the verification results can be directly displayed in the transmission tool, eliminating the steps of manually checking the table data volume and table data structure, saving verification time.

[0096] The following is a detailed description in conjunction with another embodiment.

[0097] Embodiment 2

[0098] A data transmission device between multiple data sources provided in this embodiment includes multiple implementation units, each of which corresponds to each implementation step in the above-mentioned embodiment 1.

[0099] Figure 7 is a schematic diagram of an optional data transmission device between multiple data sources according to an embodiment of the present invention. Figure 7 As shown, the data transmission device between multiple data sources may include: a connecting unit 71 , a determining unit 72 , a transmitting unit 73 , and a verifying unit 74 .

[0100] Among them, the connection unit 71 is used to receive data source information of the data source to be synchronized and the target data source input by the external terminal, and establish a communication connection between the data source to be synchronized and the target data source, wherein the data source information includes: server address, port, database type, database name, database account and password.

[0101] The determination unit 72 is used to receive the transmission mode and storage path selected by the external terminal, and determine the target transmission data based on the transmission mode, wherein the transmission mode includes: table transmission, user transmission and database transmission, and the target transmission data includes one or more target transmission files.

[0102] The transmission unit 73 exports each target transmission file from the to-be-synchronized data source to the storage path in batches, and imports the target transmission file from the storage path to the target data source when it is detected that all target transmission files are exported.

[0103] The verification unit 74 is used to perform data verification on the target transmission data in the target data source and output a verification result when it is detected that all the target transmission data are imported into the target data source.

[0104] The data transmission device between the above-mentioned multiple data sources receives the data source information of the data source to be synchronized and the target data source input by the external terminal through the connection unit 71, establishes a communication connection between the data source to be synchronized and the target data source, and then receives the transmission mode and storage path selected by the external terminal through the determination unit 72, and determines the target transmission data based on the transmission mode, wherein the target transmission data includes one or more target transmission files, and for each target transmission file, the target transmission file is exported from the data source to be synchronized to the storage path in batches through the transmission unit 73, and when it is detected that all the target transmission files are exported, the target transmission files are imported from the storage path to the target data source, and finally, when it is detected that all the target transmission data are imported into the target data source, the verification unit 74 performs data verification on the target transmission data in the target data source, outputs the verification result, and thus completes the transmission process of the target transmission data from the data source to be synchronized to the target data source.

[0105] In an embodiment of the present invention, the target transmission data includes one or more target transmission files, each target transmission file is exported and imported in batches, and the export and import process of each target transmission file is performed independently, that is, multiple target transmission files can be exported and imported at the same time. This transmission method significantly improves the transmission efficiency of the target transmission data. At the same time, the tables and fields in the target data source are updated based on the tables and fields in the data source to be synchronized, avoiding data update failures caused by differences between the tables or fields of the target data source and the data source to be synchronized, significantly reducing the error probability of data transmission, and thus solving the technical problem in the related technology that data transmission depends on overall transmission, resulting in poor transmission stability and low efficiency.

[0106] Optionally, the connection unit 71 includes: a first change module, used to obtain the configuration file of the data source to be synchronized, and change the configuration file of the data source to be synchronized based on the data source information of the data source to be synchronized and the target data source; a second change module, used to obtain the configuration file of the target data source, and change the configuration file of the target data source based on the data source information of the data source to be synchronized and the target data source; an establishment module, which establishes a communication connection between the data source to be synchronized and the target data source based on the configuration files of the data source to be synchronized and the target data source, and outputs the connection result.

[0107] Optionally, the determination unit 72 includes: a first determination module, which is used to query all tables under the data source to be synchronized when the transmission mode is transmission by table, and output all the queried tables, receive the table selected by the external terminal from all the tables, and determine the selected table as the target transmission data; a second determination module, which is used to query all users under the data source to be synchronized when the transmission mode is transmission by user, and output all the queried users, then receive the user selected by the external terminal from all the users, and determine all the tables under the selected user as the target transmission data; a third determination module, which is used to determine all tables under the data source to be synchronized as the target transmission data when the transmission mode is transmission by database.

[0108] Optionally, the transmission unit 73 includes: a judgment module, used to judge whether a target user and a target table under the target user exist in the target data source; a first creation module, used to create a target user and a target table under the target user in the target data source if the target user does not exist in the target data source; and a second creation module, used to create a target table under the target user in the target data source if the target user exists in the target data source but the target table under the target user does not exist.

[0109] Optionally, the transmission unit 73 also includes: a first detection module, used to detect whether the target transmission files have all been exported to the storage path to obtain a first detection result; an import module, used to call a pre-built import program to import the target transmission files from the storage path to the target table of the target data source when the first detection result indicates that the target transmission files have all been exported.

[0110] Optionally, the transmission unit 73 also includes: a second detection module, used to detect whether the target transmission files have been completely imported into the target data source to obtain a second detection result; a deletion module, used to delete the target transmission files in the data source to be synchronized and the storage path when the second detection result indicates that the target transmission files have been completely imported into the target data source.

[0111] Optionally, the verification unit 74 includes: a first verification module, used to compare the data volume of the target transmission data in the data source to be synchronized and the target data source, and when the comparison result indicates that the data volume is consistent, it is determined that the data verification has passed; a second verification module, used to compare the field data of each table in the target transmission data in the data source to be synchronized and the target data source, and when the comparison result indicates that the field data of all tables are consistent, it is determined that the data verification has passed.

[0112] The data transmission device between the above-mentioned multiple data sources may also include a processor and a memory. The above-mentioned connection unit 71, determination unit 72, transmission unit 73, verification unit 74, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0113] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be provided, and data transmission between multiple data sources can be performed by adjusting kernel parameters.

[0114] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory (fl ash RAM), and the memory includes at least one storage chip.

[0115] Embodiment 3

[0116] An embodiment of the present application may provide an electronic device, Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown, the electronic device may include: one or more ( Figure 8 (only one is shown) processor 802, memory 804, storage controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module and display.

[0117] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned method. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0118] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: receiving data source information of the data source to be synchronized and the target data source input by the external terminal, and establishing a communication connection between the data source to be synchronized and the target data source, wherein the data source information includes: server address, port, database type, database name, database account and password; receiving the transmission method and storage path selected by the external terminal, and determining the target transmission data based on the transmission method, wherein the transmission method includes: transmission by table, transmission by user and transmission by database, and the target transmission data includes one or more target transmission files; for each target transmission file, exporting the target transmission file from the data source to be synchronized to the storage path in batches, and when it is detected that all the target transmission files are exported, importing the target transmission file from the storage path to the target data source; when it is detected that all the target transmission data are imported into the target data source, data verification is performed on the target transmission data in the target data source, and the verification result is output.

[0119] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: obtain the configuration file of the data source to be synchronized, and change the configuration file of the data source to be synchronized based on the data source information of the data source to be synchronized and the target data source; obtain the configuration file of the target data source, and change the configuration file of the target data source based on the data source information of the data source to be synchronized and the target data source; establish a communication connection between the data source to be synchronized and the target data source based on the configuration files of the data source to be synchronized and the target data source, and output the connection result.

[0120] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: when the transmission mode is transmission by table, query all tables under the data source to be synchronized, output all queried tables, receive the table selected by the external terminal from all tables, and determine the selected table as the target transmission data; when the transmission mode is transmission by user, query all users under the data source to be synchronized, output all queried users, then receive the user selected by the external terminal from all users, and determine all tables under the selected user as the target transmission data; when the transmission mode is transmission by database, determine all tables under the data source to be synchronized as the target transmission data.

[0121] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: determine whether there is a target user and a target table under the target user in the target data source; if the target user does not exist in the target data source, create a new target user and a target table under the target user in the target data source; if the target user exists in the target data source but there is no target table under the target user, create a new target table under the target user in the target data source.

[0122] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: detect whether the target transmission files have all been exported to the storage path to obtain a first detection result; when the first detection result indicates that the target transmission files have all been exported, call a pre-built import program to import the target transmission files from the storage path to the target table of the target data source.

[0123] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: detect whether the target transmission files have been completely imported into the target data source to obtain a second detection result; when the second detection result indicates that the target transmission files have been completely imported into the target data source, delete the target transmission files in the data source to be synchronized and the storage path.

[0124] The processor can also call the information and application stored in the memory through the transmission device to perform the following steps: compare the data volume of the target transmission data in the data source to be synchronized and the target data source, and determine that the data verification has passed when the comparison result indicates that the data volume is consistent; or compare the field data of each table in the target transmission data in the data source to be synchronized and the target data source, and determine that the data verification has passed when the comparison result indicates that the field data of all tables are consistent.

[0125] By adopting the embodiment of the present application, a scheme of a data transmission method between multiple data sources is provided, by receiving the data source information of the data source to be synchronized and the target data source input by an external terminal, a communication connection between the data source to be synchronized and the target data source is established, and then by receiving the transmission mode and storage path selected by the external terminal, the target transmission data is determined based on the transmission mode, wherein the target transmission data includes one or more target transmission files, for each target transmission file, the target transmission file is exported from the data source to be synchronized to the storage path in batches, and when it is detected that all the target transmission files are exported, the target transmission file is imported from the storage path to the target data source, and finally, when it is detected that all the target transmission data are imported into the target data source, data verification is performed on the target transmission data in the target data source, and the verification result is output, thereby completing the transmission process of the target transmission data from the data source to be synchronized to the target data source, thereby achieving the purpose of significantly improving the transmission efficiency of the target transmission data and reducing the error probability of data transmission, thereby solving the technical problem in the related technology that data transmission depends on the overall transmission, resulting in poor transmission stability and low efficiency.

[0126] It can be understood by those skilled in the art that Figure 8The structure shown is for illustration only, and the electronic device may also be a terminal device such as a smart phone, a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, etc. Figure 8 The structure of the electronic device is not limited. Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 8 Different configurations shown.

[0127] A person skilled in the art may understand that all or part of the steps in the data transmission method between various multiple data sources in the above-mentioned embodiments may be completed by instructing the hardware related to the terminal device through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0128] Embodiment 4

[0129] The embodiment of the present application further provides a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the data transmission method between multiple data sources provided in the first embodiment.

[0130] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium including a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the method for transmitting data between multiple data sources of any one of the above-mentioned embodiments one.

[0131] Optionally, in this embodiment, the above storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0132] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the data transmission method between multiple data sources described in each embodiment of the present application.

[0133] The present application also provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the data transmission method between multiple data sources described in each embodiment of the present application are implemented.

[0134] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0135] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0136] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0137] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0138] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0139] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0140] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data transmission method between multiple data sources, characterized in that: Use pre-built data transfer tools, including: Receiving data source information of a data source to be synchronized and a target data source inputted by an external terminal, and establishing a communication connection between the data source to be synchronized and the target data source, wherein the data source information includes: server address, port, database type, database name, database account and password; Receiving the transmission mode and storage path selected by the external terminal, and determining target transmission data based on the transmission mode, wherein the transmission mode includes: transmission by table, transmission by user, and transmission by database, and the target transmission data includes one or more target transmission files; For each of the target transmission files, export the target transmission files from the to-be-synchronized data source to the storage path in batches, and when it is detected that all the target transmission files are exported, import the target transmission files from the storage path to the target data source; When it is detected that all the target transmission data are imported into the target data source, data verification is performed on the target transmission data in the target data source, and a verification result is output.

2. The data transmission method according to claim 1, characterized in that: The step of establishing a communication connection between the data source to be synchronized and the target data source includes: Acquire a configuration file of the data source to be synchronized, and modify the configuration file of the data source to be synchronized based on data source information of the data source to be synchronized and the target data source; Obtaining a configuration file of the target data source, and modifying the configuration file of the target data source based on data source information of the data source to be synchronized and the target data source; Based on the configuration files of the data source to be synchronized and the target data source, a communication connection is established between the data source to be synchronized and the target data source, and a connection result is output.

3. The data transmission method according to claim 1, characterized in that: The step of determining target transmission data based on the transmission mode includes: In the case where the transmission mode is transmission by table, querying all tables under the data source to be synchronized, outputting all the queried tables, receiving a table selected by the external terminal from all the tables, and determining the selected table as the target transmission data; In the case where the transmission mode is transmission by user, all users under the data source to be synchronized are queried, and all the queried users are output, and then a user selected by the external terminal from all the users is received, and all tables under the selected user are determined as the target transmission data; In the case where the transmission mode is transmission by database, all tables under the data source to be synchronized are determined as the target transmission data.

4. The data transmission method according to claim 1, characterized in that: Before exporting the target transmission file from the to-be-synchronized data source, the method further includes: Determine whether a target user and a target table under the target user exist in the target data source; If the target user does not exist in the target data source, creating the target user and a target table under the target user in the target data source; In a case where the target user exists in the target data source but a target table under the target user does not exist, a new target table is created under the target user in the target data source.

5. The data transmission method according to claim 1, characterized in that: The step of importing the target transmission file from the storage path to the target data source includes: Detecting whether all the target transmission files have been exported to the storage path to obtain a first detection result; When the first detection result indicates that the target transmission file has been completely exported, a pre-built import program is called to import the target transmission file from the storage path into the target table of the target data source.

6. The data transmission method according to claim 1, characterized in that: After all the target transmission files are imported into the target data source, the method includes: Detecting whether all the target transmission files have been imported into the target data source to obtain a second detection result; When the second detection result indicates that the target transmission file has been completely imported into the target data source, the data source to be synchronized and the target transmission file in the storage path are deleted.

7. The data transmission method according to claim 1, characterized in that: The step of performing data verification on the target transmission data in the target data source comprises: Compare the data amounts of the target transmission data in the to-be-synchronized data source and the target data source, and if the comparison result indicates that the data amounts are consistent, determine that the data verification has passed; or, The field data of each table in the target transmission data in the data source to be synchronized is compared with that in the target data source, and when the comparison result indicates that the field data of all tables are consistent, it is determined that the data verification has passed.

8. A data transmission device between multiple data sources, characterized in that: Use pre-built data transfer tools, including: A connection unit, configured to receive data source information of a data source to be synchronized and a target data source inputted by an external terminal, and establish a communication connection between the data source to be synchronized and the target data source, wherein the data source information includes: server address, port, database type, database name, database account and password; a determination unit, configured to receive a transmission mode and a storage path selected by the external terminal, and determine target transmission data based on the transmission mode, wherein the transmission mode includes: transmission by table, transmission by user, and transmission by database, and the target transmission data includes one or more target transmission files; A transmission unit, for each of the target transmission files, exports the target transmission files from the to-be-synchronized data source to the storage path in batches, and when it is detected that all the target transmission files are exported, imports the target transmission files from the storage path to the target data source; The verification unit is used to perform data verification on the target transmission data in the target data source and output a verification result when it is detected that all the target transmission data are imported into the target data source.

9. An electronic device, characterized in that: The invention comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data transmission method between multiple data sources as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for transmitting data between multiple data sources as claimed in any one of claims 1 to 7 are implemented.

Images