Data Exchange Method, System, Device and Storage Medium

Abstract

The present application discloses a data exchange method, system, device and storage medium, belonging to the field of data exchange. The method is applied to a first device installed with multiple switching engines, and the multiple switching engines respectively support different data exchange requirements, including: first selecting a switching engine that supports the target data exchange requirement from the multiple switching engines, then determining the selected switching engine as the source switching engine, and performing a data exchange task based on the source switching engine. In this way, when the user implements the task of the target data exchange requirement, the user can directly select a switching engine that supports the target data exchange requirement from the multiple switching engines through the first device, and then implement data exchange based on the selected switching engine, without the user being familiar with the reconstruction code of the switching engine and establishing a bridging channel by writing an executable script. Therefore, the technical requirements for the user can be reduced, the efficiency of data exchange can be improved, and the diverse requirements of the user for data exchange in the big data environment can be met.

Description

Technical Field

[0001] This application relates to the field of data exchange, and particularly to a data exchange method, system, device, and storage medium. Background Art

[0002] With the continuous development of informatization, the internal tasks of various industry units, institutions, or departments have gradually achieved task informatization through data exchange. Among them, in the field of database technology, data exchange refers to managing the data transmission process between databases to achieve various tasks, and data transmission refers to the process of data transmission from the source database to the target database.

[0003] In the prior art, data exchange can be achieved through an open-source bridge. The bridge, also known as a switching engine, is essentially a code collection used to establish a bridging channel between the source database and the target database and provide data exchange functions. For example, a user can download the open-source code of the source switching engine in the first device where the source database is located, download the open-source code of the target switching engine in the second device where the target database is located, and configure the operating environments of the source switching engine and the target switching engine in the first device and the second device respectively. Then, an executable script is written in the first device to call the source switching engine and the target switching engine, and a bridging channel is established between the source database and the target database by calling the source switching engine and the target switching engine to achieve the data exchange function. Among them, the open-source code of the bridge includes at least one component code, and the component code corresponds to the implemented task function. The executable script includes the address of the first device, the address of the second device, the address of the source database, the address of the target database, the method of calling the component code of the source switching engine, and the method of calling the component code of the target switching engine, etc. For example, when a user needs to establish a file transfer task, the executable script includes the address of the first device, the address of the second device, the address of the source database, the address of the target database, the method of calling the file transfer component code of the source switching engine, the method of calling the output component code of the target switching engine, etc.

[0004] However, in the above method of implementing data exchange using the open-source code of the exchange engine, it is necessary to establish a bridging channel by writing an executable script, which depends on the user's familiarity with the open-source code of the exchange engine. Therefore, the technical requirements for users are relatively high, and users spend a lot of time in configuration, resulting in low efficiency of data exchange. In addition, different exchange engines support different data exchange requirements. If different data exchange requirements are to be achieved, users need to download the open-source code of other exchange engines again in the first device and the second device, configure the operating environment, and establish a bridging channel by writing an executable script to achieve different data exchange requirements, which is not conducive to the diverse needs of users for data exchange in the big data environment. For example, if the open-source code of the exchange engine does not include the component code for real-time data exchange, there will be a problem that the real-time data exchange requirement cannot be supported. Summary of the Invention

[0005] The present application provides a data exchange method, system, device, and storage medium, which can reduce the technical requirements for users, improve the efficiency of data exchange, and meet the diverse needs of users for data exchange. The technical solutions are as follows:

[0006] In a first aspect, a data exchange method is provided. The method is applied to a first device, and the first device is installed with multiple exchange engines, and the multiple exchange engines respectively support different data exchange requirements. The method includes:

[0007] Select an exchange engine that supports the target data exchange requirement from the multiple exchange engines;

[0008] Determine the selected exchange engine as the source exchange engine;

[0009] Execute a data exchange task based on the source exchange engine.

[0010] As an example, the target data exchange requirement includes one or more of an exchange mode requirement, an operation mode requirement, and a data writing mode requirement. The exchange mode requirement is a real-time exchange mode or a non-real-time exchange mode, the operation mode is an online operation mode or an offline operation mode, and the data writing mode is a multi-threaded data writing mode or a single-threaded data writing mode.

[0011] As an example, the multiple exchange engines include a first exchange engine that supports the real-time exchange mode, and the target data exchange requirement includes an exchange mode requirement, and the exchange mode requirement is the real-time exchange mode or the non-real-time exchange mode;

[0012] The step of selecting an exchange engine that supports the target data exchange requirement from the multiple exchange engines includes:

[0013] Determine whether the exchange mode requirement is the real-time exchange mode;

[0014] If it is determined that the exchange mode requirement is the real-time exchange mode, select the first exchange engine that supports the real-time data exchange mode from the multiple exchange engines.

[0015] As an example, the multiple exchange engines further include a second exchange engine that supports the online operation mode, and the target data exchange requirement further includes an operation mode requirement, and the operation mode requirement is the online operation mode or the offline operation mode;

[0016] After determining whether the exchange mode requirement is the real-time exchange mode, it further includes:

[0017] If it is determined that the exchange mode requirement is not the real-time exchange mode, determine whether the operation mode requirement is the online operation mode;

[0018] If it is determined that the operation mode requirement is the online operation mode, select the second exchange engine that supports the online operation mode from the multiple exchange engines.

[0019] As an example, the multiple exchange engines further include a third exchange engine that supports the multi-threaded data writing mode and a fourth exchange engine that supports the single-threaded data writing mode, and the target data exchange requirement further includes a data writing mode requirement, and the data writing mode requirement is the multi-threaded data writing mode or the single-threaded data writing mode;

[0020] After determining whether the operation mode requirement is the online operation mode, it further includes:

[0021] If it is determined that the operation mode requirement is not the online operation mode, determine whether the data writing mode requirement is the multi-threaded data writing mode;

[0022] If it is determined that the data writing mode requirement is the multi-threaded data writing mode, select the third exchange engine that supports the multi-threaded data writing mode from the multiple exchange engines;

[0023] If it is determined that the data writing mode requirement is not the multi-threaded data writing mode, select the fourth exchange engine that supports the single-threaded data writing mode from the multiple exchange engines.

[0024] As an example, the multiple exchange engines include a second exchange engine that supports the online operation mode, and the target data exchange requirement includes an operation mode requirement, and the operation mode requirement is the online operation mode or the offline operation mode;

[0025] Selecting an exchange engine that supports the target data exchange requirement from multiple exchange engines includes:

[0026] Determine whether the operation mode requirement is the online operation mode;

[0027] If it is determined that the operation mode requirement is the online operation mode, select the second switching engine that supports the online operation mode from the multiple switching engines.

[0028] As an example, the multiple switching engines include a third switching engine and a fourth switching engine that support the multi-threaded data writing mode, the fourth switching engine supports the single-threaded data writing mode, the target data exchange requirement includes a data writing mode requirement, and the data writing mode requirement is the multi-threaded data writing mode or the single-threaded data writing mode;

[0029] The selecting a switching engine that supports the target data exchange requirement from the multiple switching engines includes:

[0030] Determine whether the data writing mode requirement is the multi-threaded data writing mode;

[0031] If it is determined that the data writing mode requirement is the multi-threaded data writing mode, select the third switching engine that supports the multi-threaded data writing mode from the multiple switching engines;

[0032] If it is determined that the data writing mode requirement is not the multi-threaded data writing mode, select the fourth switching engine that supports the single-threaded data writing mode from the multiple switching engines.

[0033] As an example, before determining the switching engine that supports the target data exchange requirement from the multiple switching engines, the method further includes:

[0034] Obtain a data exchange configuration file, where the data exchange configuration file includes a target data exchange requirement identifier, and the target data exchange requirement identifier is an identifier of the target data exchange requirement;

[0035] The determining a switching engine that supports the target data exchange requirement from the multiple switching engines includes:

[0036] Determine the switching engine that supports the target data exchange requirement from the multiple switching engines according to the target data exchange requirement identifier.

[0037] As an example, the data exchange configuration file further includes an execution identifier, and the execution identifier indicates an execution mode of data exchange, and the execution mode includes configuration execution and automatic execution;

[0038] Before determining, according to the identified target data exchange requirements, the exchange engines that support the target data exchange requirements from the multiple exchange engines, the method further includes:

[0039] Determine whether the execution mode indicated by the execution identifier is the automatic execution or the configured execution;

[0040] If it is determined that the execution mode indicated by the execution identifier is the automatic execution, then perform the step of determining, according to the identified target data exchange requirements, the exchange engines that support the target data exchange requirements from the multiple exchange engines.

[0041] As an example, determining the exchange engines that support the target data exchange requirements from the multiple exchange engines includes:

[0042] Detect an exchange engine selection operation;

[0043] Determine, from the multiple exchange engines, the exchange engine selected by the exchange engine selection operation as the exchange engine that supports the target data exchange requirements.

[0044] As an example, before detecting the exchange engine selection operation, the method further includes:

[0045] Obtain a data exchange configuration file, where the data exchange configuration file includes an execution identifier that indicates an execution mode of data exchange, and the execution mode includes configured execution and automatic execution;

[0046] Determine whether the execution mode indicated by the execution identifier is the automatic execution or the configured execution;

[0047] If it is determined that the execution mode indicated by the execution identifier is the configured execution, then display an exchange engine selection interface for selecting an exchange engine;

[0048] The detection of the exchange engine configuration operation includes:

[0049] Detect the exchange engine selection operation based on the exchange engine selection interface.

[0050] As an example, the multiple exchange engines include at least two of the Canal exchange engine, the NiFi exchange engine, the DataX exchange engine, and the self-developed exchange engine. The Canal exchange engine supports the real-time exchange mode, the NiFi exchange engine supports the online operation mode, the DataX exchange engine supports the multi-threaded data writing mode, the self-developed exchange engine supports the single-threaded data writing mode, and the self-developed exchange engine supports multiple types of databases.

[0051] As an example, before selecting a switching engine that supports the target data exchange requirement from multiple switching engines, the method further includes:

[0052] Obtain a data exchange configuration file, where the data exchange configuration file includes a first device address, a second device address, a source database address, and a target database address, and the second device corresponding to the second device address is installed with the multiple switching engines;

[0053] The performing a data exchange task based on the source switching engine includes:

[0054] Determine a target switching engine that matches the source switching engine from the multiple switching engines installed on the second device according to the second device address;

[0055] Establish a bridging channel between the source database and the target database according to the first device address, the second device address, the source database address, and the target database address;

[0056] Perform a data exchange task based on the bridging channel.

[0057] In a second aspect, a data exchange system is provided. The data exchange system includes a task management module and multiple bridge modules. The multiple bridge modules correspond to multiple switching engines one by one. Any one of the multiple bridge modules integrates a corresponding switching engine, and the multiple switching engines respectively support different data exchange requirements;

[0058] The task management module is configured to select a switching engine that supports the target data exchange requirement from multiple switching engines;

[0059] The task management module is further configured to determine the selected switching engine as the source switching engine;

[0060] The bridge module corresponding to the source switching engine among the multiple bridge modules is configured to perform a data exchange task based on the source switching engine.

[0061] As an example, the task management module is further configured to manage the multiple switching engines corresponding to the multiple bridge modules one by one.

[0062] As an example, the system further includes a task monitoring module, and the task monitoring module is configured to monitor the data exchange tasks performed by the multiple bridge modules.

[0063] In a third aspect, a computer device is provided. The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the above data exchange method is implemented.

[0064] In a fourth aspect, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the above data exchange method.

[0065] The beneficial effects brought by the technical solutions provided in the embodiments of the present application are as follows:

[0066] In the embodiments of the present application, a plurality of switching engines are pre-installed in the first device, and the plurality of switching engines respectively support different data exchange requirements. When performing data exchange, a switching engine that supports the target data exchange requirement can be first selected from the plurality of switching engines, and then the selected switching engine is determined as the source switching engine, and then the data exchange task is executed based on the source switching engine. In this way, if it is necessary to implement a task with a target data exchange requirement, the first device can directly select a switching engine that supports the target data exchange requirement from the plurality of switching engines, and then implement data exchange based on the selected switching engine, without the user being familiar with the refactoring code of the switching engine, nor establishing a bridging channel by writing an executable script. Therefore, the technical requirements for the user can be reduced, the data exchange efficiency can be improved, and the diverse needs of users for data exchange in a big data environment can be met. Description of the Drawings

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0068] Figure 1 is a system schematic diagram of a data exchange system provided by an embodiment of the present application;

[0069] Figure 2 is a flowchart of a data exchange method provided by an embodiment of the present application;

[0070] Figure 3 is a flowchart of a method for selecting a switching engine that supports a target data exchange requirement from a plurality of switching engines provided by an embodiment of the present application;

[0071] Figure 4 is a logical schematic diagram of configuring a switching engine provided by an example of the present application;

[0072] Figure 5 is a structural schematic diagram of a computer device provided by an embodiment of the present application. Detailed Embodiments

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the following will further describe the embodiments of this application in detail with reference to the accompanying drawings.

[0074] It should be understood that the "multiple" mentioned in this application refers to two or more. In the description of this application, unless otherwise specified, " / " means "or". For example, A / B can represent A or B. The "and / or" in this article is merely a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, to facilitate a clear description of the technical solution of this application, terms such as "first" and "second" are used to distinguish the same items or similar items with basically the same functions and roles. Those skilled in the art can understand that the terms "first" and "second" do not limit the quantity and execution order, and the terms "first" and "second" do not necessarily limit differences.

[0075] Before explaining the embodiments of this application in detail, the application scenarios of the embodiments of this application will be described first.

[0076] A data exchange method, system, device, and storage medium provided by an embodiment of this application can be applied to any scenario that needs to implement a data exchange task. For example, it can be applied to a scenario where a user needs to implement a data replication task, that is, a scenario where the data to be exchanged in the source database in the sending device needs to be copied to the target database in the receiving device. Or, it can be applied to a scenario where a user needs to implement a data migration task, that is, a scenario where the data to be exchanged in the source database in the sending device needs to be migrated to the target database in the receiving device.

[0077] Please refer to Figure 1 , Figure 1 which is a system schematic diagram of a data exchange system provided by an embodiment of this application. Among them, the first device 10 includes a data exchange system 101 and a source database 102, and the second device 20 includes a data exchange system 201 and a target database 202. Through the data exchange system 101 and the data exchange system 201, a data exchange method from the source database 102 to the target database 202 can be realized.

[0078] Among them, the first device 10 is a sending device, and the second device 20 is a receiving device. The first device 10 and the second device 20 can be computer devices or servers, etc., and the embodiments of this application do not limit this.

[0079] As Figure 1 shown, the data exchange system 101 of the first device 10 includes a task management module 1011 and multiple bridge module 1012.

[0080] Among them, the task management module 1011 is used to select a switching engine that supports the target data exchange requirements from multiple switching engines, and determine the selected switching engine as the source switching engine.

[0081] For example, if the target data exchange requirements include exchange mode requirements and the exchange mode requirement is the real-time exchange mode, the task management module 1011 can select a switching engine that supports the implementation of the exchange mode from multiple switching engines.

[0082] As an example, the task management module 101 is also used to manage the multiple switching engines corresponding to the multiple bridge modules 1012 one by one. For example, manage the data exchange policies of multiple switching engines respectively. In this way, the controllability of the data exchange process can be realized, and the problem that the data exchange process is uncontrollable in the prior art when implementing data exchange according to open source code because only the open source code is called can be solved.

[0083] Among them, the multiple bridge modules 1012 correspond to the multiple switching engines one by one. Any one of the multiple bridge modules 1012 integrates the corresponding switching engine, that is, the data exchange system 101 integrates multiple switching engines, and the multiple switching engines support different data exchange requirements respectively.

[0084] Among them, the bridge module corresponding to the source switching engine in the multiple bridge modules 1012 is used to execute the data exchange task based on the source switching engine. For the convenience of description, the bridge module corresponding to the source switching engine in the embodiments of the present application is called the source bridge module.

[0085] As an example, the source bridge module can execute the data exchange task based on the data exchange policy of the source switching engine. The data exchange policy includes one or more of a routing policy, a blocking policy, a timeout processing policy, and an alarm policy.

[0086] As an example, the source bridge module can first obtain the data to be exchanged, and then execute the data exchange task on the data to be exchanged based on the source switching engine.

[0087] Since the data exchange system 101 integrates multiple switching engines, multiple switching engines can be installed in the first device 10 by installing the data exchange system 101 in the first device 10.

[0088] In addition, any one of the multiple bridge modules 1012 is a module obtained by code refactoring and overall compilation of the corresponding switching engine. Any one of the multiple bridge modules 102 can manage the components of the corresponding switching engine. For example, the management function of the components of the switching engine can be realized by code refactoring and overall compilation of the switching engine.

[0089] In this way, when a user wants to execute a task required by the target data exchange, the data exchange system 101 in the first device 10 can select a switching engine that supports the target data exchange requirements from multiple switching engines, and then implement data exchange based on the selected switching engine. The user does not need to be familiar with the reconstruction code of the switching engine, nor does the user need to establish a bridging channel by writing an executable script. Therefore, the technical requirements for the user can be reduced, the data exchange efficiency can be improved, and the diverse requirements of the user for data exchange in the big data environment can be met.

[0090] As an example, the data exchange system 101 may further include a task monitoring module 1013, and the task monitoring module 1013 is used to monitor the data exchange tasks executed by the multiple bridge modules 1012. Among them, the monitoring of the data exchange tasks may include bridge monitoring, gateway monitoring, the transmission volume of the source database, and the reception volume of the target database, etc. Of course, the task monitoring module 1013 may perform other monitoring on the data exchange system 101, the source database 101, or the first device 10, and the embodiments of the present application do not limit this. For example, the task monitoring module 1013 monitors the total exchange volume of the first device 10.

[0091] For example, the task monitoring module 1013 monitors the transmission volume of the source database 102 and monitors the reception volume of the target database 202 in the first device 20. When it is determined that the transmission volume and the reception volume are not the same, a prompt message is sent to prompt the user that the exchange task fails.

[0092] Among them, the data exchange system 201 of the second device 20 includes a task management module 2011 and multiple bridge modules 2012. The task management module 2011 is similar in function to the task management module 1011 of the data exchange system 101 of the first device 10, and the multiple bridge modules 2012 are similar in function to the task management module 1012 of the data exchange system 101 of the first device 10, and will not be elaborated here.

[0093] As an example, the data exchange system 201 of the second device 20 may further include a task monitoring module 2013, and the task monitoring module 2013 is similar in function to the task monitoring module 1013 of the data exchange system 101 of the first device 10, and will not be elaborated here.

[0094] Since the data exchange system 201 integrates multiple switching engines, multiple switching engines can be installed in the second device 20 by installing the data exchange system 201 in the second device 20.

[0095] As an example, after the task management module 1011 of the first device 10 determines the source switching engine, it can also determine a target switching engine that matches the source switching engine from multiple switching engines installed in the second device 20. Then, through the source switching engine and the target switching engine, a bridging channel is established between the source database 102 and the target database 202, and the data to be exchanged is transmitted based on the bridging channel to execute the data exchange task.

[0096] It should be noted that in the embodiments of the present application, only the first device 10 is taken as the sending device where the source database 102 is located, and the second device 20 is taken as the receiving device where the target database 202 is located as an example for illustration. When performing other data exchange tasks, the first device 10 can also be the receiving device, and the second device 20 can also be the sending device. The embodiments of the present application do not limit this.

[0097] In addition, in the embodiments of the present application, only the first device 10 and the second device 20 are taken as different devices for illustration. It can be understood that the first device 10 and the second device 20 can also be the same device. In this case, data exchange between different databases in the same device can be realized. Of course, the source database 102 and the target database 202 can also be the same database. In this case, exchange of different fields in the same database in the same device can be realized.

[0098] Next, a data exchange method provided by the embodiments of the present application will be described in detail.

[0099] Please refer to Figure 2 , Figure 2 which is a flowchart of a data exchange method provided by the embodiments of the present application. As shown in Figure 2 , the data exchange method can be applied to the first device. Multiple switching engines are installed in the first device, and the multiple switching engines respectively support different data exchange requirements. For example, it can be applied to the data exchange system of the first device shown in Figure 1 . The data exchange method may include the following steps:

[0100] Step 201, the first device selects a switching engine that supports the target data exchange requirement from multiple switching engines.

[0101] It should be noted that the first device is the device where the source database is located, that is, the device where the data to be exchanged is located.

[0102] For example, please refer to Figure 1 , the first device is installed with a data exchange system. The data exchange system includes a task management module and multiple bridge modules. The task management module of the data exchange system of the first device can select a switching engine that supports the target data exchange requirement from multiple switching engines.

[0103] Among them, multiple switching engines respectively support different data switching requirements. The target data switching requirements include one or more of a switching method requirement, an operation method requirement, and a data writing method requirement. The switching method requirement can be a real-time switching method or a non-real-time switching method, the operation method can be an online operation method or an offline operation method, and the data writing method can be a multi-threaded data writing method or a single-threaded data writing method.

[0104] That is to say, the target data switching requirements include at least one requirement, and a switching engine can be selected from multiple switching engines according to at least one requirement.

[0105] In addition, the order of selecting a switching engine according to at least one requirement may not be fixed. For example, if the target data switching requirements include a switching method requirement and an operation method requirement, a switching engine can be selected from at least one switching engine according to the switching method requirement first, or a switching engine can be selected from at least one switching engine according to the operation method requirement first; or a switching engine can be excluded from at least one switching engine according to the switching method requirement first, and then a switching engine other than the excluded switching engine can be selected from at least one switching engine according to the operation method requirement; or a switching engine can be excluded from at least one switching engine according to the switching method requirement first, and then a switching engine other than the excluded switching engine can be selected from at least one switching engine according to the operation method requirement. The embodiments of the present application do not limit this.

[0106] As an example, among multiple engines, there is a first switching engine that supports the real-time switching method. The target data switching requirements include a switching method requirement, and the switching method requirement is a real-time switching method or a non-real-time switching method. Selecting a switching engine that supports the target data switching requirements from multiple switching engines can be achieved through the following steps:

[0107] Step 2011, determine whether the switching method requirement is a real-time switching method.

[0108] Among them, determining whether the target data switching requirement is a real-time switching method is to determine whether the requirement of the user's task of implementing data switching is real-time. Among them, the real-time switching method means obtaining the data to be switched from the source database in real time and sending the data to be switched to the target database in real time. The non-real-time switching method means that after obtaining the data to be switched, the data to be switched is stored first, and then the stored data to be switched is sent to the target database at an appropriate time. The non-real-time switching method can effectively reduce the probability of data switching congestion.

[0109] Step 2012, if it is determined that the switching method requirement is a real-time switching method, select the first switching engine that supports the real-time data switching method from multiple switching engines.

[0110] Among them, since the first switching engine supports the real-time switching mode and can meet the real-time target data switching requirements, the first switching engine can be selected from multiple switching engines to perform the data switching task of the real-time switching mode that supports the target data switching requirements based on the first switching engine.

[0111] Among them, after selecting the first switching engine, it is also possible to determine whether the first switching engine supports other requirements of the target data switching requirements. For example, if the data writing method requirement of the target data switching requirement is the multi-threaded data writing method, it is also possible to determine whether the first switching engine supports the multi-threaded data writing method. If not, the components of the first switching engine are configured so that the first switching engine supports the multi-threaded data writing method.

[0112] For example, the task management module of the data switching system of the first device can configure the components of the first switching engine. To make the first switching engine support the multi-threaded data writing method, the task management module can add a method component that calls the multi-threaded data writing component in the third switching engine to the components of the first switching engine, so that the first switching engine can support the multi-threaded data writing method.

[0113] Among them, if it is determined that the switching mode requirement is not the real-time switching mode, any switching engine can be selected from multiple switching engines, and any switching engine can also be the first switching engine.

[0114] For example, after determining that the switching mode requirement is not the real-time switching mode, that is, after determining that the switching mode requirement is the non-real-time switching mode, other target data switching requirements can be used to continue to select a switching engine other than the first switching engine from multiple switching engines, so as to meet the diverse needs of users for data switching in the big data environment.

[0115] As an example, after determining that the switching mode requirement is not the real-time switching mode, selecting a switching engine from multiple switching engines can also be achieved through the following three implementation methods:

[0116] The first implementation method: The multiple switching engines also include a second switching engine that supports the online operation mode, and the target data switching requirements also include an operation mode requirement, and the operation mode requirement is the online operation mode or the offline operation mode. Then, after determining that the switching mode requirement is not the real-time switching mode, it is also possible to determine whether the operation mode requirement is the online operation mode. If it is determined that the operation mode requirement is the online operation mode, the second switching engine that supports the online operation mode is selected from multiple switching engines.

[0117] Among them, the online operation mode refers to the unified online (remote) monitoring of data exchange tasks. The offline operation mode refers to the inability to unify the online (remote) monitoring of data exchange tasks. For example, the exchange engine of the first device can monitor the data exchange tasks executed by the first device and can also monitor the data exchange tasks executed by the second device. In this case, it can be considered that the exchange engine of the first device supports the online operation mode.

[0118] In addition, after determining that the operation mode requirement is not the online operation mode, it is also possible to continue to select other exchange engines from multiple exchange engines except the first exchange engine and the second exchange engine.

[0119] The second implementation method: The multiple exchange engines also include a third exchange engine that supports the multi-threaded data writing method. The target data exchange requirement also includes the data writing method requirement, and the data writing method requirement is the multi-threaded data writing method or the single-threaded data writing method. Then, after determining that the exchange method requirement is not the real-time exchange method, it is also possible to determine whether the data writing method requirement is the multi-threaded data writing method. If it is determined that the data writing method requirement is the multi-threaded data writing method, then select the third exchange engine that supports the multi-threaded data writing method from the multiple exchange engines.

[0120] Among them, the multi-threaded data writing method means that multiple threads can be started at the same time to execute data exchange tasks, so that multiple data exchange tasks can be executed at the same time, and efficient and massive data exchange can be achieved. The single-threaded data writing method means that only one thread can be started at the same time to execute the data exchange task, and the data exchange task can only be sequentially executed in one thread. For example, the multi-threaded data writing method converts one database type into multiple other database types at the same time, and the single-threaded data writing method can only convert one database type into other database types at the same time. If multiple database type conversions are to be achieved, they need to be converted one by one in sequence.

[0121] Among them, after determining that the data writing method requirement is not the multi-threaded data writing method, it is also possible to select any one of the multiple exchange engines. Any one of the exchange engines can be an exchange engine other than the first exchange engine and the third exchange engine among the multiple exchange engines.

[0122] The third implementation method: The target data exchange requirement also includes the operation mode requirement and the data writing method requirement. The operation mode is the online operation mode or the offline operation mode, and the data writing method is the multi-threaded data writing method or the single-threaded data writing method. After determining that the exchange method requirement is not the real-time exchange method, it is also possible to select an exchange engine that supports the target data exchange requirement by determining whether the operation mode requirement is the online operation mode and determining whether the data writing method requirement is the multi-threaded data writing method.

[0123] It should be noted that the specific implementation process of the third implementation method will be described in detail in the following Figure 3 embodiments, and will not be elaborated here in the embodiments of the present application.

[0124] It should be noted that the above three implementation methods are only used as an example and are used as a limitation for selecting a switching engine from multiple switching engines.

[0125] As an example, multiple switching engines include a second switching engine. The second switching engine supports an online operation mode. The target data exchange requirement includes an operation mode requirement, and the operation mode requirement is an online operation mode or an offline operation mode. Then, selecting a switching engine that supports the target data exchange requirement from multiple switching engines can be achieved through the following steps:

[0126] Step 2013, determine whether the operation mode requirement is an online operation mode.

[0127] Step 2014, if it is determined that the operation mode requirement is an online operation mode, then select the second switching engine that supports the online operation mode from multiple switching engines.

[0128] Among them, after selecting the second switching engine, the task management module can also determine whether the second switching engine supports other requirements of the target data exchange requirement. For example, if the data writing method requirement of the target data exchange requirement is a multi-threaded data writing method, it can also be determined whether the second switching engine supports the multi-threaded data writing method. If not, the components of the second switching engine are configured so that the second switching engine supports the multi-threaded data writing method.

[0129] For example, the task management module of the data exchange system of the first device can configure the components of the second switching engine. To make the second switching engine support the multi-threaded data writing method, the task management module can add a method component for calling the multi-threaded data writing component in the first switching engine to the components of the second switching engine, so that the second switching engine can support the online operation mode.

[0130] It should be noted that if it is determined that the switching mode requirement is not a real-time switching mode, any switching engine can be selected from multiple switching engines, and any switching engine can also be the second switching engine.

[0131] For example, other switching engines other than the second switching engine can continue to be selected from multiple switching engines according to other requirements of the target data exchange requirement. The specific implementation can refer to the above steps 2011 and 2012, and will not be elaborated here.

[0132] As an example, multiple switching engines include a third switching engine that supports multi-threaded data writing mode and a fourth switching engine that supports single-threaded data writing mode. The target data exchange requirements include data writing mode requirements, and the data writing mode requirements are multi-threaded data writing mode or single-threaded data writing mode. Selecting a switching engine that supports the target data exchange requirements from multiple switching engines can be achieved through the following steps:

[0133] Step 2015, determine whether the data writing mode requirement is multi-threaded data writing mode.

[0134] Step 2016, if it is determined that the data writing mode requirement is multi-threaded data writing mode, then select the third switching engine that supports multi-threaded data writing mode from multiple switching engines.

[0135] Among them, after selecting the third switching engine, the task management module of the data exchange system of the first device can also determine whether the third switching engine supports other requirements of the target data exchange requirements. For example, if the operation mode requirement of the target data exchange requirement is online operation mode, it can also be determined whether the third switching engine supports the online operation mode. If not, the components of the third switching engine are configured so that the first switching engine supports the online operation mode.

[0136] Step 2017, if it is determined that the data writing mode requirement is not multi-threaded data writing mode, then select the fourth switching engine that supports single-threaded data writing mode from multiple switching engines.

[0137] Among them, after selecting the fourth switching engine, the task management module of the data exchange system of the first device can also determine whether the fourth switching engine supports other requirements of the target data exchange requirements. For example, if the operation mode requirement of the target data exchange requirement is online operation mode, it can also be determined whether the fourth switching engine supports the online operation mode. If not, the components of the fourth switching engine are configured so that the first switching engine supports the online operation mode.

[0138] As an example, the target data exchange requirements can also be determined by other means. For example, the target data exchange requirements include data writing mode requirements. The data exchange configuration file includes the amount of data to be exchanged and the writing time to determine the data writing mode requirements. For example, if the amount of data to be exchanged is large and the writing time is short, the data writing mode is determined to be multi-threaded data writing mode.

[0139] It should be noted that in the embodiments of the present application, steps 2011 and 2012, steps 2013 and 2014, and steps 2015, 2016 and 2017 can be combined arbitrarily to select a switching engine that supports the target data exchange requirements from multiple switching engines. For example, as followsFigure 3 An embodiment provides a method for selecting a switching engine that supports target data exchange requirements from multiple switching engines.

[0140] In addition, the target data exchange requirements include multiple requirements. When selecting a switching engine according to the multiple requirements of the target data exchange requirements, if the execution order of the multiple requirements is different, the selected switching engine is different.

[0141] As an example, the multiple switching engines may include at least two of a Canal switching engine, a NiFi switching engine, a DataX switching engine, and a self-developed switching engine.

[0142] Among them, the Canal switching engine is an incremental data exchange tool based on the incremental log parsing of MySQL (a relational database management system) open-sourced by Alibaba. Its advantage is that it supports real-time exchange methods. The NiFi switching engine is an easy-to-use, powerful, and reliable data exchange tool open-sourced abroad. Its advantage is that it supports online operation methods and realizes online monitoring of data exchange tasks. The DataX switching engine is an offline massive data synchronization tool open-sourced by Alibaba. Its advantage is that it supports multi-threaded task execution operations, that is, it supports multi-threaded data writing methods.

[0143] Among them, the self-developed switching engine is a switching engine independently developed by the applicant of the embodiment of the present application, which supports visualization and multiple types of databases. The self-developed switching engine is a powerful and flexible tool for data extraction, conversion, and pushing. By embedding a script engine, a conversion engine, a process engine, and a scheduling engine, it improves the work efficiency of heterogeneous system interconnection, avoids the interface development between heterogeneous systems and gateways, and can achieve the interconnection and intercommunication with heterogeneous systems with "zero coding". The self-developed switching engine provides a graphical configuration interface, and the configuration of data exchange policies is completed through the graphical configuration interface. In addition, the self-developed switching engine can support multiple database types, such as MySQL, Oracle (a relational database management system), SQLserver (a relational database management system), DB2 (distributed database), ODPS (Open Data Processing Service), HDFS (Hadoop Distributed File System), Hive (a data warehouse tool based on Hadoop), HBase (a distributed storage system), Dameng Database, and RenDaJinCang Database, etc.

[0144] As an example, the first switching engine can be a Canal switching engine, the second switching engine can be a NiFi switching engine, the third switching engine can be a DataX switching engine, and the fourth switching engine can be a self-developed switching engine.

[0145] It should be noted that the Canal switching engine, NiFi switching engine, and DataX switching engine in the embodiments of the present application are switching engines obtained by performing code refactoring and overall compilation on the open-source code of the corresponding switching engines.

[0146] It should also be noted that the data exchange implemented directly using the open-source code of the bridge in the prior art does not meet the requirements of independent controllability in the information technology application innovation industry. For example, it is impossible to determine whether operations unrelated to data exchange, such as operations that endanger data security, are performed during the process of implementing data exchange using the open-source code of the bridge. Since the Canal switching engine, NiFi switching engine, or DataX switching engine in the embodiments of the present application are switching engines that are respectively subjected to code refactoring and overall compilation, they can meet the requirements of independent controllability in the information technology application innovation industry.

[0147] For example, the open-source code of the Canal switching engine only supports MySQL and does not support other database types. After the open-source code of the Canal switching engine is refactored and overall compiled in the embodiments of the present application, the obtained Canal switching engine not only supports MySQL but also supports database types such as Oracle, DM Database, and KingbaseES, increasing the database types supported by the Canal switching engine.

[0148] For example, the open-source code of the NiFi switching engine is not localized, and the operation threshold is relatively high. After the open-source code of the NiFi switching engine is refactored and overall compiled in the embodiments of the present application, the obtained NiFi switching engine not only realizes the functions of the original open-source code but also provides localization services.

[0149] For example, the open-source code of the DataX switching engine supports HDFS, Hive, HBase, ODPS, etc., but does not support domestic DM Database and KingbaseES, and it only supports running in script execution mode, which has relatively high technical requirements for users. After the open-source code of the DataX switching engine is refactored and overall compiled in the embodiments of the present application, the obtained DataX switching engine not only retains support for the original database types but also supports DM Database and KingbaseES, and at the same time provides a visual interface, reducing the technical requirements for users.

[0150] It should be noted that by refactoring the code of the exchange engine and performing overall compilation, the components of multiple exchange engines can be managed to meet the requirements of independent control in the domestic information technology application innovation industry.

[0151] Of course, multiple exchange engines may also include other exchange engines, which are not limited in the embodiments of this application. For example, multiple exchange engines also include the Kettle exchange engine.

[0152] Among them, the target data exchange requirement can be determined according to the identifier of the target data exchange requirement in the data exchange configuration file, or can be independently determined by the user according to the needs. For example, by triggering an exchange engine selection operation to select an exchange engine that meets the target data exchange requirement.

[0153] As an example, the exchange engine that supports the target data exchange requirement can be determined through the following two implementation methods:

[0154] The first implementation method: Obtain a data exchange configuration file including the identifier of the target data exchange requirement, determine the target data exchange requirement through the identifier of the target data exchange requirement, and thus determine the exchange engine that supports the target data exchange requirement from multiple exchange engines according to the identifier of the target data exchange requirement.

[0155] Among them, the identifier of the target data exchange requirement is the identifier of the target data exchange requirement, which is used to indicate the target data exchange requirement. For example, the identifier of the target data exchange requirement includes one or more of the identifier of the exchange method requirement, the identifier of the operation method requirement, and the identifier of the data writing method requirement.

[0156] In a possible implementation manner, the data exchange configuration file further includes an execution identifier, and the execution identifier indicates the execution method of the data exchange. The execution method includes configuration execution and automatic execution. Before determining the exchange engine that supports the target data exchange requirement from multiple exchange engines according to the identifier of the target data exchange requirement, it is also possible to first determine whether the execution method indicated by the execution identifier is automatic execution or configuration execution. If it is determined that the execution method indicated by the execution identifier is automatic execution, then perform the step of determining the exchange engine that supports the target data exchange requirement from multiple exchange engines according to the identifier of the target data exchange requirement. That is, when the execution method is automatic execution, the first device can determine the exchange engine that supports the target data exchange requirement from multiple exchange engines according to the identifier of the target data exchange requirement.

[0157] For example, the data exchange configuration file can be obtained from other systems in the first device. For example, if the first device is installed with a resource directory system, the task management module can obtain the data exchange configuration file generated by the resource target system. Among them, the resource directory system is an asset management system for managing data resources, and the resource directory system includes a publishing module, a generating module, an approval authorization management module, and a retrieval and subscription module. Among them, the publishing module is used to publish the data resources of the database, the generating module is used to generate the data exchange configuration file, the approval authorization management module is used to receive the application request for initiating the data exchange task and obtain the approval result of the data exchange task, and the retrieval and subscription module is used to retrieve and subscribe to the data resources published by the publishing module and initiate the application request for the data exchange task.

[0158] Specifically, the first device and the second device are respectively installed with a data exchange system and a resource directory system. The publishing module of the first device publishes the data resources of the source database. The retrieval and subscription module of the second device retrieves and subscribes to the data resources published by the publishing module according to the requirements (i.e., retrieves and subscribes to the data to be exchanged), determines the target data exchange requirement identifier corresponding to the target data exchange requirement, and initiates an application request for the data exchange task to the approval authorization management module of the first device. The approval authorization management module of the first device receives the application request for the data exchange task, obtains the approval result. If it is determined that the approval result is passed, the generating module of the first device will automatically generate a data exchange configuration file, and the data exchange configuration file includes the target data exchange requirement identifier.

[0159] As an example, the data exchange configuration file can also be generated according to the data exchange configuration operation. For example, first detect the data exchange configuration operation, and generate the data exchange configuration file according to the target data exchange requirement identifier configured by the data exchange configuration operation.

[0160] For example, the task management module of the data exchange system of the first device can detect the data exchange configuration operation based on the data exchange configuration interface. For example, the first device includes a display screen, and the first device displays the data exchange configuration interface through the display screen. The data exchange configuration interface is used to configure the data exchange configuration file, that is, to configure the target data exchange requirement identifier. The task management module can detect the data exchange configuration operation based on the data exchange configuration interface.

[0161] Among them, the data exchange configuration operation can be triggered by the user, and the operation type of the data exchange configuration operation can be a click operation or a press operation, etc. The embodiments of the present application do not limit this.

[0162] It should be noted that for the first implementation method, the target data exchange requirement identifier may include one or more of the exchange method requirement identifier, the operation method requirement identifier, and the data writing method requirement identifier. Therefore, after obtaining the data exchange configuration file including the target data exchange requirement identifier, the exchange method requirement, the operation method requirement, or the data writing method requirement can be determined according to the exchange method requirement identifier, the operation method requirement identifier, or the data writing method requirement, and then the exchange engine that supports the target data exchange requirement can be determined from multiple exchange engines according to the exchange method requirement, the operation method requirement, or the data writing method requirement. Specifically, the specific implementation of determining the exchange engine that supports the target data exchange requirement from multiple exchange engines according to the exchange method requirement, the operation method requirement, or the data writing method requirement can refer to step 2011 and step 2012, or refer to step 2013 and step 2014, or refer to step 2015, step 2016, and step 2017, or refer to the following Figure 3 Example.

[0163] As an example, the target data exchange requirement identifier may be the target data exchange requirement name or the target data exchange requirement ID, etc., and the embodiments of the present application do not limit this. For example, the target data exchange requirement identifier includes the exchange method requirement identifier, and the exchange method requirement identifier may be the exchange method requirement name or the exchange method requirement ID, etc. For example, the exchange method requirement ID is a code composed of the numbers 0 and 1.

[0164] In this way, after the user performs a simple operation of retrieving and subscribing to the data to be exchanged, or after the user performs a simple data exchange configuration operation, the task management module of the data management system can automatically generate a data exchange configuration file including the target data exchange requirement identifier, and select an exchange engine that supports the target data exchange requirement from multiple exchange engines according to the target data exchange requirement identifier. Further, the selected exchange engine implements the data exchange task of the target data exchange requirement, without the user being familiar with the refactoring code of the exchange engine and establishing a bridging channel by writing an executable script. Therefore, the technical requirements for the user can be reduced, the data exchange efficiency can be improved, and the diverse requirements of the user for data exchange in the big data environment can be met.

[0165] The second implementation method: Detect the exchange engine selection operation, and determine the exchange engine selected by the exchange engine selection operation as the exchange engine that supports the target data exchange requirement, that is, determine the exchange engine selected by the exchange engine selection operation from multiple exchange engines as the exchange engine that supports the target data exchange requirement.

[0166] Among them, before detecting the exchange engine selection operation, a data exchange configuration file may be obtained first. The data exchange configuration file includes an execution identifier, and the execution identifier indicates the execution mode of data exchange. The execution mode includes configured execution and automatic execution. Then, it is determined whether the execution mode indicated by the execution identifier is automatic execution or configured execution. If it is determined that the execution mode indicated by the execution identifier is configured execution, the exchange engine selection operation is detected.

[0167] As an example, the execution identifier may be an execution mode name, an execution mode code, etc., and the embodiments of the present application do not limit this.

[0168] As an example, the task management module of the data exchange system of the first device may detect the exchange engine selection operation based on the exchange engine selection interface. For example, the first device includes a display screen, and the first device can display the exchange engine selection interface through the display screen. The exchange engine selection interface is used to select an exchange engine. The task management module may detect the exchange engine selection operation based on the exchange engine selection interface.

[0169] The exchange engine selection operation may be triggered by the user. The operation type of the exchange engine selection operation may be a click operation, a press operation, etc., and the embodiments of the present application do not limit this.

[0170] It should be noted that at least one exchange engine that supports the target data exchange requirement can be selected according to the user-triggered exchange engine configuration operation, or at least one exchange engine that supports the target data exchange requirement can be selected according to the target data exchange requirement identifier in the data exchange configuration file, without the user being familiar with the refactoring code of the exchange engine and without establishing a bridging channel by writing an executable script. Therefore, the technical requirements for the user can be reduced, the data exchange efficiency can be improved, and the diverse data exchange needs of users in the big data environment can be met.

[0171] Step 202: Determine the selected exchange engine as the source exchange engine.

[0172] Among them, the source exchange engine refers to the exchange engine used to send the data of the source database to the target database. Correspondingly, the target exchange engine refers to the exchange engine used to receive the data in the source database and store the data in the source database into the target database.

[0173] For example, the task management module of the data exchange system of the first device selects the source exchange engine. The first device is the device where the source data database is located, and the source exchange engine is one of the multiple exchange engines installed in the first device, and the source exchange engine supports the target data exchange requirement.

[0174] Step 203: Execute the data exchange task based on the source exchange engine.

[0175] For example, the source bridge module corresponding to the source switching engine of the data switching system of the first device obtains the data to be switched, and performs data switching on the data to be switched based on the source switching engine.

[0176] As an example, the data to be switched can be obtained according to the source database address and the data to be switched identifier. For example, the data indicated by the data to be switched identifier in the source database is used as the data to be switched. Among them, the source database address can be the source database name or source database code, etc., and the data to be switched identifier can be the table name in the source database or the position of the field in the table, etc. The data switching configuration file can include the source database address and the data to be switched identifier.

[0177] In addition, the data switching configuration file can also include the second device address. The task management module of the data switching system of the first device can also determine the target switching engine matching the source switching engine from multiple switching engines installed in the second device according to the second device address. Then, a bridging channel is established between the source database and the target database, and the data switching task is executed through this bridging channel. For example, data switching is performed on the data to be switched in the source database through this bridging channel, that is, the data to be switched in the source database is sent to the target database.

[0178] For example, if the source switching engine is the first switching engine installed in the first device, the target switching engine matching the source switching engine is the first switching engine installed in the second device.

[0179] As an example, the data switching system integrates multiple switching engines, and the user can pre-install multiple switching engines in the second device by installing the data switching system in the second device.

[0180] In addition, multiple switching engines may not be pre-installed in the second device. After the task management module of the data switching system of the first device determines the source switching engine, it can also install the target switching engine matching the source switching engine in the second device remotely according to the second device address, without offline configuration and deployment, improving the working efficiency of data switching.

[0181] Among them, after the task management module of the data switching system of the first device determines the target switching engine, it can also determine the bridging channel between the source database and the target database, so that the source bridge module can execute the data switching task based on the bridging channel, that is, send the data to be switched from the source database to the target database based on the bridging channel to complete the data switching.

[0182] As an example, the data exchange configuration file includes a first device address, a second device address, a source database address, and a target database address. A bridging channel can be established between the source database and the target database according to the first device address, the second device address, the source database address, and the target database address.

[0183] Among them, the first device address and the second device address can be computer device IP (Internet Protocol) addresses or server IP addresses, or computer device gateway IP addresses or server gateway IP addresses. The embodiments of the present application do not make limitations in this regard.

[0184] Among them, the first device address and the second device address can be the same or different. The first device address and the second device address are in the same local area network or may not be in the same local area network.

[0185] Among them, the target database address can be a target database name or a target database ID (Identity Document), etc. The embodiments of the present application do not make limitations in this regard.

[0186] As an example, a bridging channel can also be established between the source database and the target database according to the first device address, the second device address, the source database address, the target database address, the user identifier, and the password identifier. The bridging channel can be identified through the user identifier and the password identifier, improving the security of data exchange.

[0187] As an example, the task management module of the data exchange system of the first device can also manage multiple exchange engines in the first device.

[0188] For example, manage the data exchange policy of the source exchange engine, so that the source exchange engine executes the data exchange task based on the data exchange policy, which can realize the controllability of the data exchange process and solve the problem that the data exchange process is uncontrollable in the prior art when implementing data exchange according to open source code because it is only a call to the open source code.

[0189] As an example, the data exchange policy includes one or more of a routing policy, a blocking policy, a timeout processing policy, and an alarm policy. For example, the task management module of the data exchange system of the first device sets the data exchange policy of the source exchange engine as the default policy of the exchange engine. The default policies of different exchange engines can be the same or different. Or, the user can also configure the data exchange policy of the source exchange engine. The embodiments of the present application do not make limitations in this regard.

[0190] As an example, when the source switching engine performs a data switching task, the data switching system can also monitor the data switching task performed by the source switching engine. For example, the data switching system includes a task monitoring module, which can monitor the tasks performed by the source switching engine to achieve the monitoring of the data switching process, facilitating the user to promptly discover data switching failures and find the reasons for data switching failures.

[0191] As an example, when the source target engine performs data switching on the data to be switched based on a data switching policy, other components of the source target engine can also be called to implement other functions. For example, during the process of implementing the switching of the data to be switched from the source data to the target data, data format conversion, database type conversion, data cleaning, etc. also need to be implemented.

[0192] For example, the source target engine can determine the components called during the data switching process according to the data to be switched. For example, if the format of the data to be switched is.doc (document), and the format defined for storage in the target database is.pdf (Portable Document Format), then when the source target engine performs the data switching task, it also calls the data format conversion component of the source switching engine to convert the.doc - formatted data to be switched into the.pdf format and then store it in the target database.

[0193] As an example, when the source target engine performs data switching on the data to be switched based on a data switching policy, components of other switching engines except the source switching engine among the multiple switching engines in the first device can also be called to support the target data switching requirements.

[0194] In the embodiments of the present application, the first device is pre - installed with multiple switching engines, and the multiple switching engines respectively support different data switching requirements. When performing data switching, the switching engine that supports the target data switching requirements can be first selected from the multiple switching engines, and then the selected switching engine is determined as the source switching engine, and then the data switching task is performed based on the source switching engine. In this way, if a task with target data switching requirements is to be implemented, the switching engine that supports the target data switching requirements can be directly selected from the multiple switching engines by the first device, and then data switching is implemented based on the selected switching engine, without the user having to be familiar with the refactored code of the switching engine, nor having to establish a bridging channel by writing an executable script. Therefore, the technical requirements for the user can be reduced, the data switching efficiency can be improved, and the diverse requirements of users for data switching in a big data environment can be met.

[0195] Please refer to Figure 3 , Figure 3 which is a flowchart of a method for selecting a switching engine that supports target data switching requirements from multiple switching engines provided by the embodiments of the present application.Figure 3 The method provided by the embodiment can be applied to a first device, for example, to a data exchange system in the first device, and is executed by a task management module of the data exchange system of the first device. Figure 3 The method provided by the embodiment. The method includes:

[0196] Step 301, the first device determines whether the exchange mode requirement is a real-time exchange mode.

[0197] It should be noted that in the embodiments of the present application, the target data exchange requirements include an exchange mode requirement, an operation mode requirement, and a data writing mode requirement. The exchange mode requirement is a real-time exchange mode or a non-real-time exchange mode, the operation mode is an online operation mode or an offline operation mode, and the data writing mode is a multi-threaded data writing mode or a single-threaded data writing mode.

[0198] In addition, in the embodiments of the present application, multiple exchange engines include a first exchange engine that supports the real-time data exchange mode, a third exchange engine that supports the multi-threaded data writing mode, a third exchange engine that supports the multi-threaded data writing mode, and a fourth exchange engine that supports the single-threaded data writing mode.

[0199] Step 302, if the first device determines that the exchange mode requirement is a real-time exchange mode, it selects the first exchange engine that supports the real-time data exchange mode from multiple exchange engines.

[0200] For the specific descriptions of steps 302 and 302, please refer to Figure 2 Steps 2011 and 2012 of the above embodiments.

[0201] As an example, after selecting the first exchange engine, the task management module can also determine whether the first exchange engine supports other requirements of the target data exchange requirements. For example, if the operation mode requirement of the target data exchange requirement is an online operation mode, it can also determine whether the first exchange engine supports the online operation mode. If not, the components of the first exchange engine are configured so that the first exchange engine supports the online operation mode.

[0202] As an example, the first exchange engine can be a Canal exchange engine.

[0203] Step 303, if the first device determines that the exchange mode requirement is not a real-time exchange mode, it determines whether the operation mode requirement is an online operation mode.

[0204] Wherein, the exchange mode requirement is not a real-time exchange mode, that is, the exchange mode requirement is a non-real-time exchange mode.

[0205] Among them, after determining that the operation mode requirement is not the real-time exchange mode, any one of the multiple exchange engines can be selected, and any one of the exchange engines can be the first exchange engine.

[0206] For example, according to the operation mode requirement of the target data exchange requirement, an exchange engine other than the first exchange engine is selected from the multiple exchange engines, or according to the data writing mode requirement of the target data exchange requirement, an exchange engine other than the first exchange engine is selected from the multiple exchange engines. In the embodiments of the present application, only an example is given in which after determining that the operation mode requirement is not the online operation mode, an exchange engine is selected from the multiple exchange engines according to the operation mode requirement.

[0207] Step 304, if the first device determines that the operation mode requirement is the online operation mode, it selects a second exchange engine that supports the online operation mode from the multiple exchange engines.

[0208] In this way, according to the exchange mode requirement being the non-real-time exchange mode and the operation mode requirement being the online operation mode, a second exchange engine can be selected from the multiple exchange engines. Since the second exchange engine supports the non-real-time exchange mode and the online operation mode, the target data exchange requirements of non-real-time and online operations can be realized. Therefore, a second exchange engine can be selected from the multiple exchange engines to perform data exchange tasks that support the non-real-time exchange mode and the online operation mode of the target data exchange requirements based on the second exchange engine.

[0209] Among them, after selecting the second exchange engine, the task management module can also determine whether the second exchange engine supports other requirements of the target data exchange requirements. For example, if the data writing mode requirement of the target data exchange requirement is the multi-threaded data writing mode, it can also be determined whether the second exchange engine supports the multi-threaded data writing mode. If not, the components of the second exchange engine are configured so that the second exchange engine supports the multi-threaded data writing mode.

[0210] As an example, the second exchange engine can be the NiFi exchange engine.

[0211] Step 305, if the first device determines that the operation mode requirement is not the online operation mode, it determines whether the data writing mode requirement is the multi-threaded data writing mode.

[0212] Among them, after determining that the operation mode requirement is not the online operation mode, that is, after determining that the operation mode requirement is the offline operation mode, any one of the multiple exchange engines can be continuously selected according to other target data exchange requirements. Any one of the exchange engines can be the first exchange engine or the second exchange engine.

[0213] For example, according to the data writing method requirement of the target data exchange requirement, select a switching engine other than the first switching engine and the second switching engine from multiple switching engines.

[0214] Step 306: If the first device determines that the data writing method requirement is a multi-threaded data writing method, select a third switching engine that supports the multi-threaded data writing method from multiple switching engines.

[0215] In this way, according to the switching method requirement being a non-real-time switching method, the operation method requirement being an offline operation method, and the data writing method requirement being a multi-threaded data writing method, select a third switching engine from multiple switching engines. Since the third switching engine supports the non-real-time switching method, the offline operation method, and the multi-threaded data writing method, the target data exchange requirement of non-real-time, online operation, and multi-threaded data writing can be achieved. Therefore, a third switching engine can be selected from multiple switching engines to perform data exchange tasks that support the non-real-time switching method, the offline operation method, and the multi-threaded data writing method based on the third switching engine.

[0216] As an example, the third switching engine can be a DataX switching engine.

[0217] Step 307: If the first device determines that the data writing method requirement is not a multi-threaded data writing method, select a fourth switching engine that supports the single-threaded data writing method from multiple switching engines.

[0218] In this way, according to the switching method requirement being a non-real-time switching method, the operation method requirement being an offline operation method, and the data writing method requirement being a single-threaded data writing method, select a fourth switching engine from multiple switching engines. Since the fourth switching engine supports the non-real-time switching method, the offline operation method, and the single-threaded data writing method, the target data exchange requirement of non-real-time, online operation, and single-threaded data writing can be achieved. Therefore, a fourth switching engine can be selected from multiple switching engines to perform data exchange tasks that support the non-real-time switching method, the offline operation method, and the single-threaded data writing method based on the fourth switching engine.

[0219] As an example, the fourth switching engine can be a self-developed switching engine.

[0220] It should be noted that the embodiments of the present application are only examples, and other methods can also be used to select a method that supports the target data exchange requirements from multiple exchange engines. For example, if the target data exchange requirements include more requirements, those skilled in the art can easily think that the execution orders of more requirements are different, and the exchange engines selected from multiple exchange engines are different. Since one exchange engine is selected from multiple exchange engines and the target data exchange requirements supported by multiple exchange engines are different, the diversity of data exchange requirements of users for data exchange in the big data environment can be satisfied. In addition, the diversity of data exchange requirements of users for data exchange in the big data environment can be further satisfied by mutual calls of components between multiple exchange engines.

[0221] For ease of understanding, the data exchange method described in the embodiments of the present application is exemplified by multiple exchange engines including Canal exchange engine, NiFi exchange engine, DataX exchange engine, and self-developed exchange engine. Among them, the Canal exchange engine supports real-time data exchange mode, the NiFi exchange engine supports multi-threaded data writing mode, the DataX exchange engine supports multi-threaded data writing mode, and the self-developed exchange engine supports single-threaded data writing mode. Please refer to Figure 4 , Figure 4 which is a logical schematic diagram of configuring an exchange engine provided by an example of the present application. As Figure 4 shown, the logical process of configuring an exchange engine may include the following steps:

[0222] Step 401, the first device obtains a data exchange configuration file.

[0223] Among them, the data exchange configuration file includes an execution identifier and a target data exchange requirement identifier.

[0224] The target data exchange requirement identifier includes an exchange mode requirement identifier, an operation mode requirement identifier, and a data writing mode requirement identifier. The exchange mode requirement identifier indicates the exchange mode requirement, and the exchange mode requirement is a real-time exchange mode or a non-real-time exchange mode; the operation mode requirement identifier indicates the operation mode requirement, and the operation mode requirement is an online operation mode or an offline operation mode; the data writing mode requirement identifier indicates the data writing mode requirement, and the data writing mode requirement is a multi-threaded data writing mode or a single-threaded data writing mode.

[0225] The execution identifier indicates the execution mode of data exchange, and the execution mode includes configuration execution and automatic execution.

[0226] Step 402, the first device determines whether the execution mode indicated by the execution identifier is automatic execution.

[0227] Among them, determining whether the execution mode indicated by the execution identifier is automatic execution is to determine whether the execution mode is automatic execution or configured execution.

[0228] Step 403, if the first device determines that the execution mode indicated by the execution identifier is not automatic execution, it detects the switching engine selection operation, determines the switching engine selected by the switching engine selection operation from multiple switching engines, and then jumps to step 411 to execute the step of performing the data exchange task based on the selected switching engine.

[0229] Among them, if it is determined that the execution mode indicated by the execution identifier is not automatic execution, it means that its execution mode is manual execution. In this case, the user can independently select a switching engine from multiple switching engines according to the requirements. The selected switching engine can be a Canal switching engine, a NiFi switching engine, a DataX switching engine, or a self-developed switching engine.

[0230] Step 404, if the first device determines that the execution mode indicated by the execution identifier is automatic execution, it determines whether the switching mode is a real-time switching mode.

[0231] Among them, if it is determined that the execution mode indicated by the execution identifier is not automatic execution, it means that its execution mode is automatic execution. In this case, the first device can select a switching engine from multiple switching engines according to the target data exchange requirements.

[0232] Step 405, if the first device determines that the switching mode requirement is a real-time switching mode, it selects a Canal switching engine from multiple switching engines, and then jumps to step 411 to execute the step of performing the data exchange task based on the selected switching engine.

[0233] Step 406, if the first device determines that the switching mode requirement is not a real-time switching mode, it determines whether the operation mode requirement is an online operation mode.

[0234] Among them, if it is determined that the switching mode requirement is not a real-time switching mode, it is to determine that the switching mode requirement is a non-real-time switching mode.

[0235] Step 407, if the first device determines that the operation mode requirement is an online operation mode, it selects a NiFi switching engine from multiple switching engines, and then jumps to step 411 to execute the step of performing the data exchange task based on the selected switching engine.

[0236] Step 408, if the first device determines that the operation mode requirement is not an online operation mode, it determines whether the data writing mode requirement is a multi-threaded data writing mode.

[0237] Among them, if it is determined that the operation mode requirement is not an online operation mode, it is to determine that the operation mode requirement is an offline operation mode.

[0238] Step 409, if the first device determines that the data writing method requirement is the multi-threaded data writing method, select the DaTaX exchange engine from multiple exchange engines, and then jump to Step 411 to execute the step of performing the data exchange task based on the selected exchange engine.

[0239] Step 410, if the first device determines that the data writing method requirement is not the multi-threaded data writing method, select the self-developed exchange engine from multiple exchange engines, and then jump to Step 411 to execute the step of performing the data exchange task based on the selected exchange engine.

[0240] Among them, if it is determined that the data writing method requirement is not the multi-threaded data writing method, that is, it is determined that the data writing method requirement is the single-threaded data writing method, the first device selects the self-developed exchange engine that supports single-threaded writing from multiple exchange engines.

[0241] Step 411, the first device performs the data exchange task based on the selected exchange engine.

[0242] Among them, the exchange engines selected in the above Steps 403, 405, 407, 409 and 410 are the source exchange engines in the above Figure 2 embodiment.

[0243] As an example, the data exchange configuration file further includes the first device address, the second device address, the source database address and the target database address. Before Step 411, the first device can also determine the target exchange engine that matches the source exchange engine from multiple exchange engines installed on the second device according to the second device address. Then, according to the first device address, the second device address, the source database address and the target database address, a bridging channel between the source database and the target database is established, and the data exchange task is performed through this bridging channel.

[0244] It should be noted that the embodiments of the present application are only used as an example, and other methods can also be used to select the method that supports the target data exchange requirement from multiple exchange engines. For example, if the target data exchange requirement includes more requirements, those skilled in the art can easily think that the execution order of more requirements is different, and the exchange engines selected from multiple exchange engines are different. Since one exchange engine is selected from multiple exchange engines and the target data exchange requirements supported by multiple exchange engines are different, the diversity of the data exchange requirements of users in the big data environment can be satisfied. In addition, the diversity of the data exchange requirements of users in the big data environment can be further satisfied by mutual calls of components between multiple exchange engines.

[0245] In the embodiments of the present application, a first device is pre-installed with multiple switching engines, and the multiple switching engines respectively support different data switching requirements. When performing data switching, it is possible to first obtain a data switching configuration file, and determine whether the first device autonomously selects a switching engine from the multiple switching engines or the user selects a switching engine from the multiple switching engines according to the data switching configuration file. If the first device autonomously selects a switching engine from the multiple switching engines, the first device selects a switching engine from the multiple switching engines according to the target data switching requirement identifier in the data switching configuration file. If the user selects a switching engine from the multiple switching engines, the first device can detect the switching engine selection operation triggered by the user and select the switching engine selected by the switching engine selection operation from the multiple switching engines. In this way, if you want to implement a task with target data switching requirements, you can directly select a switching engine that supports the target data switching requirements from the multiple switching engines through the first device, or select a switching engine that supports the target data switching requirements from the multiple switching engines through the switching engine selection operation triggered by the user, and then implement data switching based on the selected switching engine. There is no need for the user to be familiar with the refactored code of the switching engine, nor is it necessary to establish a bridging channel by writing an executable script. Therefore, the technical requirements for the user can be reduced, the efficiency of data switching can be improved, and the diverse needs of users for data switching in the big data environment can be met.

[0246] Figure 5 is a schematic structural diagram of a computer device provided by an embodiment of the present application. As Figure 5 shown, the computer device includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the above Figure 2 embodiment, the above Figure 3 embodiment, and the steps of the method provided by the above Figure 4 embodiment.

[0247] Among them, the computer device can be the first device and the second device in the above Figure 1 embodiment, the above Figure 2 embodiment, the above Figure 3 embodiment, and the above Figure 4 embodiment. The computer device can be a general computer device or a dedicated computer device. In specific implementation, the computer device can be a desktop computer, a portable computer, a network server, a handheld computer, a mobile phone, a tablet computer, a wireless terminal device, a communication device, or an embedded device. The embodiments of the present application do not limit the type of the computer device. Those skilled in the art can understand, Figure 5The computer device is only an example and does not limit the computer device. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0248] The processor 501 may be a central processing unit (CPU), or the processor 501 may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0249] In some embodiments, the memory 502 may be an internal storage unit of the computer device, such as the hard disk or memory of the computer device. In other embodiments, the memory 502 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Further, the memory 502 may also include both the internal storage unit and the external storage device of the computer device. The memory 502 is used to store an operating system, application programs, a boot loader, data, and other programs, etc. The memory 502 may also be used to temporarily store data that has been output or will be output.

[0250] An embodiment of the present application also provides a computer device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. When the processor executes the computer program, the steps in any of the above method embodiments are implemented.

[0251] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0252] An embodiment of the present application provides a computer program product, which when running on a computer, causes the computer to execute the steps in any of the above method embodiments.

[0253] When the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above method embodiments of the present application, a computer program can be used to instruct relevant hardware to complete. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable medium can at least include: any entity or device capable of carrying the computer program code to the photographing device / terminal device, recording medium, computer memory, ROM (Read-Only Memory), RAM (Random Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc. The computer-readable storage medium mentioned in the present application can be a non-volatile storage medium, in other words, it can be a non-transitory storage medium.

[0254] It should be understood that all or part of the steps of implementing the above embodiments can be achieved by software, hardware, firmware or any combination thereof. When implemented using software, it can be achieved in the form of a computer program product in whole or in part. The computer program product includes one or more computer instructions. The computer instructions can be stored in the above computer-readable storage medium.

[0255] In the above embodiments, the descriptions of each embodiment have their own focuses. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0256] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0257] In the embodiments provided in the present application, it should be understood that the disclosed device / computer device and method can be implemented in other ways. For example, the device / computer device embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling or direct coupling or communication connection between each other can be through some interfaces. The indirect coupling or communication connection of the device or unit can be in electrical, mechanical or other forms.

[0258] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0259] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data exchange method, characterized in that, The method is applied to a first device, which is installed with a plurality of switching engines, and the plurality of switching engines respectively support different data exchange requirements. The method includes: Select a switching engine that supports the target data exchange requirement from the plurality of switching engines. Before determining the switching engine that supports the target data exchange requirement from the plurality of switching engines, the method further includes: obtaining a data exchange configuration file, where the data exchange configuration file includes a target data exchange requirement identifier, and the target data exchange requirement identifier is an identifier of the target data exchange requirement; Determining the switching engine that supports the target data exchange requirement from the plurality of switching engines includes: determining, according to the target data exchange requirement identifier, the switching engine that supports the target data exchange requirement from the plurality of switching engines. The data exchange configuration file further includes an execution identifier, and the execution identifier indicates an execution manner of data exchange, and the execution manner includes configured execution and automatic execution; before determining, according to the target data exchange requirement identifier, the switching engine that supports the target data exchange requirement from the plurality of switching engines, the method further includes: determining whether the execution manner indicated by the execution identifier is the automatic execution or the configured execution; if it is determined that the execution manner indicated by the execution identifier is the automatic execution, then perform the step of determining, according to the target data exchange requirement identifier, the switching engine that supports the target data exchange requirement from the plurality of switching engines; Determine the selected switching engine as the source switching engine; Execute a data exchange task based on the source switching engine.

2. The method according to claim 1, characterized in that, The target data exchange requirement includes one or more of an exchange manner requirement, an operation manner requirement, and a data writing manner requirement. The exchange manner requirement is a real-time exchange manner or a non-real-time exchange manner, the operation manner is an online operation manner or an offline operation manner, and the data writing manner is a multi-threaded data writing manner or a single-threaded data writing manner.

3. The method according to claim 1, characterized in that The plurality of switching engines include a first switching engine that supports the real-time exchange manner, and the target data exchange requirement includes an exchange manner requirement, and the exchange manner requirement is the real-time exchange manner or the non-real-time exchange manner; Selecting a switching engine that supports the target data exchange requirement from the plurality of switching engines includes: Determine whether the exchange manner requirement is the real-time exchange manner; If it is determined that the exchange manner requirement is the real-time exchange manner, then select the first switching engine that supports the real-time data exchange manner from the plurality of switching engines.

4. The method according to claim 3, characterized in that, The plurality of switching engines further include a second switching engine that supports the online operation manner, and the target data exchange requirement further includes an operation manner requirement, and the operation manner requirement is the online operation manner or the offline operation manner; After determining whether the exchange manner requirement is the real-time exchange manner, it further includes: If it is determined that the exchange manner requirement is not the real-time exchange manner, then determine whether the operation manner requirement is the online operation manner; If it is determined that the operation mode requirement is the online operation mode, select the second switching engine that supports the online operation mode from the multiple switching engines.

5. The method according to claim 4, wherein The multiple switching engines further include a third switching engine that supports the multi-threaded data writing mode and a fourth switching engine that supports the single-threaded data writing mode. The target data exchange requirement further includes a data writing mode requirement, and the data writing mode requirement is the multi-threaded data writing mode or the single-threaded data writing mode. After determining whether the operation mode requirement is the online operation mode, it further includes: If it is determined that the operation mode requirement is not the online operation mode, determine whether the data writing mode requirement is the multi-threaded data writing mode. If it is determined that the data writing mode requirement is the multi-threaded data writing mode, select the third switching engine that supports the multi-threaded data writing mode from the multiple switching engines. If it is determined that the data writing mode requirement is not the multi-threaded data writing mode, select the fourth switching engine that supports the single-threaded data writing mode from the multiple switching engines.

6. The method according to claim 1, wherein The multiple switching engines include a second switching engine that supports the online operation mode. The target data exchange requirement includes an operation mode requirement, and the operation mode requirement is the online operation mode or the offline operation mode. Selecting a switching engine that supports the target data exchange requirement from the multiple switching engines includes: Determining whether the operation mode requirement is the online operation mode. If it is determined that the operation mode requirement is the online operation mode, select the second switching engine that supports the online operation mode from the multiple switching engines.

7. The method according to claim 1, wherein The multiple switching engines include a third switching engine and a fourth switching engine that support the multi-threaded data writing mode. The fourth switching engine supports the single-threaded data writing mode. The target data exchange requirement includes a data writing mode requirement, and the data writing mode requirement is the multi-threaded data writing mode or the single-threaded data writing mode. Selecting a switching engine that supports the target data exchange requirement from the multiple switching engines includes: Determining whether the data writing mode requirement is the multi-threaded data writing mode. If it is determined that the data writing mode requirement is the multi-threaded data writing mode, select the third switching engine that supports the multi-threaded data writing mode from the multiple switching engines. If it is determined that the data writing mode requirement is not the multi-threaded data writing mode, select the fourth switching engine that supports the single-threaded data writing mode from the multiple switching engines.

8. The method according to any one of claims 1 to 7, characterized in that, The multiple switching engines include at least two of the Canal switching engine, the NiFi switching engine, the DataX switching engine, and the self-developed switching engine. The Canal switching engine supports the real-time exchange mode. The NiFi switching engine supports the online operation mode. The DataX switching engine supports the multi-threaded data writing mode. The self-developed switching engine supports the single-threaded data writing mode, and the self-developed switching engine supports multiple types of databases.

9. The method according to any one of claims 1-7, characterized in that, Before selecting an exchange engine that supports the target data exchange requirements from multiple exchange engines, the method further includes: Obtain a data exchange configuration file, where the data exchange configuration file includes a first device address, a second device address, a source database address, and a target database address, and the second device corresponding to the second device address is installed with the multiple exchange engines; The performing the data exchange task based on the source exchange engine includes: Determine a target exchange engine that matches the source exchange engine from the multiple exchange engines installed on the second device according to the second device address; Establish a bridging channel between the source database and the target database according to the first device address, the second device address, the source database address, and the target database address; Perform the data exchange task based on the bridging channel.

10. A data exchange method, characterized in that, The method is applied to a first device, and the first device is installed with multiple exchange engines, and the multiple exchange engines respectively support different data exchange requirements. The method includes: Select an exchange engine that supports the target data exchange requirements from multiple exchange engines. Determining an exchange engine that supports the target data exchange requirements from multiple exchange engines includes: detecting an exchange engine selection operation; determining the exchange engine selected by the exchange engine selection operation from the multiple exchange engines as the exchange engine that supports the target data exchange requirements. Before detecting the exchange engine selection operation, the method further includes: obtaining a data exchange configuration file, where the data exchange configuration file includes an execution identifier, and the execution identifier indicates an execution manner of data exchange, and the execution manner includes configured execution and automatic execution; determining whether the execution manner indicated by the execution identifier is the automatic execution or the configured execution; if it is determined that the execution manner indicated by the execution identifier is the configured execution, then display an exchange engine selection interface, and the exchange engine selection interface is used to select an exchange engine; the detecting the exchange engine configuration operation includes: detecting the exchange engine selection operation based on the exchange engine selection interface; Determine the selected exchange engine as the source exchange engine; Perform the data exchange task based on the source exchange engine.

11. The method according to claim 10, wherein The multiple exchange engines include at least two of a Canal exchange engine, a NiFi exchange engine, a DataX exchange engine, and a self-developed exchange engine. The Canal exchange engine supports a real-time exchange manner, the NiFi exchange engine supports an online operation manner, the DataX exchange engine supports a multi-threaded data writing manner, the self-developed exchange engine supports a single-threaded data writing manner, and the self-developed exchange engine supports multiple types of databases.

12. A data exchange system, characterized in that, The data exchange system includes a task management module and multiple bridge modules. The multiple bridge modules correspond to the multiple exchange engines one by one. Any one of the multiple bridge modules integrates a corresponding exchange engine, and the multiple exchange engines respectively support different data exchange requirements; The task management module is used to select a switching engine that supports the target data exchange requirements from multiple switching engines. Before determining the switching engine that supports the target data exchange requirements from the multiple switching engines, a data exchange configuration file is obtained. The data exchange configuration file includes a target data exchange requirement identifier, and the target data exchange requirement identifier is an identifier of the target data exchange requirements. Determining the switching engine that supports the target data exchange requirements from the multiple switching engines includes: determining the switching engine that supports the target data exchange requirements from the multiple switching engines according to the target data exchange requirement identifier. The data exchange configuration file further includes an execution identifier, and the execution identifier indicates an execution mode of data exchange. The execution mode includes configured execution and automatic execution. Before determining the switching engine that supports the target data exchange requirements from the multiple switching engines according to the target data exchange requirement identifier, it is determined whether the execution mode indicated by the execution identifier is the automatic execution or the configured execution. If it is determined that the execution mode indicated by the execution identifier is the automatic execution, then the step of determining the switching engine that supports the target data exchange requirements from the multiple switching engines according to the target data exchange requirement identifier is executed. The task management module is further used to determine the selected switching engine as the source switching engine. The bridge module corresponding to the source switching engine in the multiple bridge modules is used to execute the data exchange task based on the source switching engine.

13. A computer device, characterized in that, The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the method described in any one of claims 1 to 11 is implemented.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the method described in any one of claims 1 to 11 is implemented.

Images