Heterogeneous data source online data exchange method and device, equipment and medium

Abstract

The invention relates to the technical field of big data, and discloses a heterogeneous data source online data exchange method, device and equipment and a medium, the method comprises the following steps: mapping to-be-exchanged data of a data table in a source database into a general data type conforming to a target database; according to the structure of a target table in the target database, a backup table is newly built, and the structure of the backup table is the same as that of the target table; copying the to-be-exchanged data to the standby table according to the general data type; and performing partition exchange on the standby table and the target table so as to load data in the standby table into the target table. According to the method, the to-be-exchanged data is copied into the standby table, the loading process does not influence external services of the target database, and the standby table does not need actions such as index checking, so that the data loading efficiency can be improved; in addition, the application target database can be switched on the second level, and the pain point caused by batch data exchange can be solved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, device, equipment and medium for online data exchange of heterogeneous data sources. Background Art

[0002] In the digital age, data application scenarios cover all aspects of business operations and management. In terms of corresponding technical implementation, a variety of technology platforms have emerged for different scenarios. In terms of data storage, there are big databases represented by Hive, Spark, Starrocks, etc., and traditional databases represented by Mysql, Oracle, Tdsql, etc. Big databases and traditional databases often need to be exchanged between different data platforms.

[0003] In the existing technology, heterogeneous data sources are equipped with corresponding data exchange tools, and the efficiency of data exchange is also a common problem. How to maintain loading efficiency without being affected by the growth rate of business data, and to ensure the effectiveness of external services while loading data as much as possible. Summary of the invention

[0004] The present invention provides a method, device, equipment and medium for online data exchange of heterogeneous data sources, so as to solve the technical problems in the prior art that data exchange efficiency between heterogeneous databases is low and external services are affected.

[0005] In a first aspect, a method for online data exchange of heterogeneous data sources is provided, comprising:

[0006] Map the data to be exchanged in the data table in the source database to the common data type that conforms to the target database;

[0007] According to the structure of the target table in the target database, a new backup table is created, wherein the structure of the backup table is the same as that of the target table;

[0008] According to the general data type, copying the data to be exchanged to the standby table;

[0009] Partition exchange is performed on the standby table and the target table to load data in the standby table into the target table.

[0010] Furthermore, the step of mapping the data to be exchanged in the data table in the source database to a common data type that conforms to the target database includes:

[0011] Determine the type of the target database, if the target database is a relational database, obtain the common data type of the target table, the common data type includes a primary key, a field and a data type;

[0012] Determine whether the data to be exchanged has a single primary key; if the data to be exchanged has multiple primary keys, establish a mapping relationship between the row key of the target table and the multiple primary keys;

[0013] The multiple primary keys of the data to be exchanged are used as row keys of the target table, and the data to be exchanged are mapped to the general data type.

[0014] Furthermore, the backup table is a bak table, and the steps of creating a new backup table according to the structure of the target table in the target database include:

[0015] If the source database is a distributed database, a non-partitioned backup table is established, and the backup table corresponds to a distribution key, and the distribution key is used to indicate the location and distribution mode of the data to be exchanged in the source database;

[0016] Otherwise, create a non-partitioned, index-free backup table.

[0017] Furthermore, the step of copying the data to be exchanged to the standby table according to the general data type includes:

[0018] The Sqoop tool is used to read the data to be exchanged in the source database in sequence, and the data read each time is loaded into the corresponding row of the standby table.

[0019] Furthermore, the partition exchange of the standby table and the target table comprises the following steps:

[0020] Determine a partition operation instruction according to the type of the target database;

[0021] According to the partition operation instruction, perform a partition operation on the target table and the backup table;

[0022] Verify whether the partition operation is successful, and delete the backup table after success.

[0023] Furthermore, the source database is a large database.

[0024] Furthermore, before the step of mapping the data to be exchanged in the data table of the source database into a common data type that conforms to the target database, the step further includes:

[0025] The data to be exchanged is cleaned to remove duplicate data and delete blank data.

[0026] In a second aspect, a device for online data exchange of heterogeneous data sources is provided, comprising:

[0027] A mapping module, used to map the data to be exchanged in the data table in the source database into a common data type that conforms to the target database;

[0028] A new creation module, used for creating a new backup table according to the structure of the target table in the target database, wherein the structure of the backup table is the same as that of the target table;

[0029] A backup module, used for copying the data to be exchanged to the standby table according to the general data type;

[0030] The exchange module is used to perform partition exchange on the standby table and the target table to load the data in the standby table into the target table.

[0031] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned method for online data exchange of heterogeneous data sources when executing the computer program.

[0032] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for online data exchange of heterogeneous data sources are implemented.

[0033] In the scheme implemented by the above-mentioned heterogeneous data source online data exchange method, device, equipment and medium, when data exchange is required between the source database and the target database, the data to be exchanged in the data table of the source database is first mapped to a common data type that conforms to the target database; then, according to the structure of the target table in the target database, a backup table with the same structure as the target table is established; then, according to the common data type, the data to be exchanged is copied to the backup table; finally, the backup table and the target table are partitioned and exchanged to load the data in the backup table into the target table.

[0034] In an embodiment of the present invention, the data to be exchanged is first copied to a spare table. Since the loading process does not affect the external services of the target database and since the spare table does not need to check indexes and other actions, the data loading efficiency can be improved. In addition, the target database of the application can be switched in seconds, which can solve the pain points caused by batch data exchange, save labor costs, improve the batch scheduling function of the platform, and at the same time ensure the normal operation of the system and enable business operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0036] Figure 1It is a schematic diagram of an application environment of an online data exchange method for heterogeneous data sources in one embodiment of the present invention;

[0037] Figure 2 It is a flow chart of a method for online data exchange of heterogeneous data sources in one embodiment of the present invention;

[0038] Figure 3 for Figure 2 A schematic diagram of a process of S10;

[0039] Figure 4 for Figure 2 A flow chart of step S40;

[0040] Figure 5 It is a structural schematic diagram of an online data exchange device for heterogeneous data sources in one embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0042] Figure 7 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The embodiment of the present invention provides a method for online data exchange of heterogeneous data sources, which can be applied in the following aspects: Figure 1 In an application environment, the client communicates with the server through a network. The server can receive a user instruction through the client, the user instruction includes a source database, data to be exchanged, a target database and a target table, and the server maps the data to be exchanged in the data table of the source database to a general data type that conforms to the target database; according to the structure of the target table in the target database, a new backup table is created, and the structure of the backup table is the same as that of the target table; according to the general data type, the data to be exchanged is copied to the backup table; and the partitions of the backup table and the target table are exchanged to load the data in the backup table into the target table.

[0045] In a scheme implemented by a method for online data exchange of heterogeneous data sources provided by the present application, when data exchange is required between a source database and a target database, the data to be exchanged in the data table in the source database is first mapped to a common data type that conforms to the target database; then, according to the structure of the target table in the target database, a backup table with the same structure as the target table is established; then, according to the common data type, the data to be exchanged is copied to the backup table; finally, the backup table and the target table are partitioned and exchanged to load the data in the backup table into the target table.

[0046] In an embodiment of the present invention, the data to be exchanged is first copied to a spare table. Since the loading process does not affect the external services of the target database and since the spare table does not need to check indexes and other actions, the data loading efficiency can be improved. In addition, the target database of the application can be switched in seconds, which can solve the pain points caused by batch data exchange, save labor costs, improve the batch scheduling function of the platform, and at the same time ensure the normal operation of the system and enable business operations.

[0047] The client may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server may be implemented by an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0048] See also Figure 2 As shown, a flow chart of a method for online data exchange of heterogeneous data sources provided by an embodiment of the present invention includes the following steps:

[0049] S10: Mapping the data to be exchanged in the data table in the source database into a common data type that conforms to the target database;

[0050] The embodiments of the present invention involve a scheme for online data exchange between a source database and a target database, wherein the source database and the target database are heterogeneous databases, wherein the source database is generally a big database, which is also called a big data repository or a big data platform. It is a system for storing, managing, and analyzing large-scale data sets, and has the characteristics of massive storage, distributed architecture, support for a variety of data types, and real-time processing capabilities. Specifically, a big database can store PB-level or even EB-level data volumes, and can accommodate structured, semi-structured, and unstructured data from various sources. For example, user behavior data of Internet companies, real-time data generated by sensors, text and image data on social media, etc. Common big data systems include Hive, Spark, and Starrocks.

[0051] Big databases are often used in the financial and medical fields. In the financial field, big databases are used for risk assessment, fraud detection, market forecasting, etc. For example, banks assess customers' credit risks by analyzing their transaction records, credit history and other data; securities institutions use big data to analyze market trends and make investment decisions. The medical industry uses big databases to store and analyze electronic medical records, medical images, genetic data, etc. to assist in disease diagnosis, treatment plan formulation and medical research. For example, by analyzing a large amount of electronic medical record data, the potential pattern of the disease and the correlation between the treatment effect can be discovered.

[0052] The target database is generally a traditional database, which is a database based on a relational model and consists of a two-dimensional table and its connections. Its main features include data structuring, data consistency, data independence, and support for complex queries. Traditional databases ensure data consistency and integrity through transaction processing and constraint mechanisms. For example, in scenarios such as financial transaction systems and corporate financial systems that have high requirements for data consistency, bank transfer services must ensure the atomicity, consistency, isolation, and persistence of transactions. The transaction processing capabilities of traditional databases can ensure that these requirements are met.

[0053] Big databases and traditional databases have their own advantages in different aspects. The two complement each other and provide more comprehensive data management solutions for various application scenarios. In this process, data exchange between big databases and traditional databases is often required. For example, in the financial field, it is often necessary to introduce customer data from external channels such as social media and e-commerce platforms and store these data in big databases. These data can supplement customer behavior information outside traditional financial scenarios, such as customer consumption habits, hobbies, social relationships, etc. Banks have structured data such as customer basic information, account information, and transaction records, which are stored in traditional relational databases. By analyzing these data, we can understand the customer's basic financial behavior, such as deposits, withdrawals, transfers, loan repayments, etc., so as to evaluate the customer's basic characteristics such as credit rating and risk preference. By exchanging heterogeneous data sources and integrating financial data in traditional databases and external data in big databases, we can build a more complete customer portrait. For example, we can judge the customer's credit status based on his past loan repayment record, which provides a reference for subsequent credit business.

[0054] When online data exchange of heterogeneous data sources is required, first search for the data to be exchanged in the data table in the source database. You can use index-based search methods, full table scan and partition scan methods. When there is an index for the data to be exchanged, select the index-based search method; when the source database does not store a large amount of data, select the full table scan method; when the source database stores a large amount of data, select the partition scan method.

[0055] As can be seen from the above description, the data formats stored in the source database are various, including numbers, text, pictures, etc. Therefore, it is necessary to map the data to be exchanged in the data table to a general data type that conforms to the target database. The general data type can generally be: for numeric types, corresponding to integer types and decimal types; for character types, corresponding to fixed-length character types and variable-length character types; for date and time types, stored in a specific format, such as YYYY-MM-DD, where "YYYY" represents the year, "MM" represents the month, and "DD" represents the date. The type of each specific numeric value, character, etc. can be determined according to the actual situation, and the embodiment of the present invention does not specifically limit this.

[0056] As an implementation manner, before the step of mapping the data to be exchanged in the data table of the source database to a common data type that conforms to the target database, the step further includes:

[0057] The data to be exchanged is cleaned to remove duplicate data and delete blank data.

[0058] In this embodiment, before mapping the data to be exchanged, it is necessary to clean the data to be exchanged, remove duplicate data, and delete blank data, so that the remaining data to be exchanged are meaningful, and after removing redundant data, the amount of data can be reduced, thereby improving data loading efficiency.

[0059] In some embodiments, Figure 3 As shown, in step S10, the data to be exchanged in the data table of the source database is mapped to a common data type that conforms to the target database, and the steps include:

[0060] S11: Determine the type of the target database. If the target database is a relational database, obtain the common data type of the target table. The common data type includes a primary key, a field, and a data type.

[0061] In the embodiment of the present invention, the type of the target database is first determined. Whether the target database is a relational database can be determined from aspects such as data structure, operation mode and data constraints. When it is determined that the target database is a relational database, the general data type of the target table is obtained.

[0062] In the embodiment of the present invention, the general data type may also include three basic information: primary key, field and data type. Among them, the primary key can ensure that each record in the table is unique to avoid data duplication. The primary key can be regarded as the ID of each record; a field is a column in a table, which is used to store data of a specific type. Each field has a name and data type to describe the data content stored in the field; the data type is a specification for specifying the type and format of the data stored in the field. Different data types have different storage methods, value ranges and operation methods, such as integer type, decimal type, character type, date type and Boolean type.

[0063] S12: Determine whether the data to be exchanged has a single primary key. If the data to be exchanged has multiple primary keys, establish a mapping relationship between the row key of the target table and the multiple primary keys.

[0064] Most database management tools provide the function of viewing table structure. Through these tools, you can intuitively see the table column information, constraints, etc. These tools can determine whether the target table has a single primary key. If it has a single primary key, a mapping relationship between the row key of the data to be exchanged and the multiple primary keys is established, and the row key of the data to be exchanged is used to uniquely represent the data to be exchanged.

[0065] If the target table has a single primary key, directly execute step S13.

[0066] For example, if the data must be unique on a single attribute, then the table may be a single primary key table. If the data needs to be unique on a combination of multiple attributes, then the table may be a multiple primary key table. For example, in a student course selection table, if each student is required to have only one course selection record for each course, then the combination of the student number and the course number may be the primary key of the table, that is, multiple primary key tables.

[0067] S13: Using the multiple primary keys of the data to be exchanged as the row keys of the target table, and mapping the data to be exchanged into the general data type.

[0068] The row key of the data to be exchanged is used as the primary key, and the data to be exchanged is mapped to a common data type, that is, the data to be exchanged is mapped to data that complies with the rules of the common data type.

[0069] S20: creating a backup table according to the structure of the target table in the target database, wherein the structure of the backup table is the same as that of the target table;

[0070] Then, according to the structure of the target table in the target database, a new backup table is created in the target database, and the structure of the backup table is exactly the same as that of the target table. Common table structures include: columns, rows, indexes, table constraints and table relationships, among which the column name is used to uniquely identify each column in the table, and is usually descriptive and can clearly express the data content stored in the column. Each row in the table represents a specific record or entity. For example, in the student information table, each row represents the information of a student. The values ​​of each column in the row are combined to describe the specific characteristics of the record. An index is a data structure used to improve the query performance of a database table. It can quickly locate a specific row in a table without scanning the entire table. Table constraints include primary key constraints, unique constraints, non-null constraints, and foreign key constraints. Table relationships include one-to-one relationships, one-to-many relationships, and many-to-many relationships.

[0071] As an implementation manner, the backup table is a bak table, and the steps of creating a new backup table according to the structure of the target table in the target database include:

[0072] If the source database is a distributed database, a non-partitioned backup table is established, and the backup table corresponds to a distribution key, and the distribution key is used to indicate the location and distribution mode of the data to be exchanged in the source database;

[0073] Otherwise, create a non-partitioned, index-free backup table.

[0074] In the embodiment of the present invention, the backup table is a bak table, which can be regarded as a temporary table. In the database, a temporary table is a special type of table, which is mainly used to temporarily store data to meet specific query or processing requirements. A temporary table is created to temporarily store data during a specific database operation. It does not exist in the database for a long time like a regular table, but can be automatically deleted after use or manually deleted as needed.

[0075] Specifically, when the source database is a distributed database, a backup table is established. The backup table is configured with a distribution key, which indicates the location and distribution mode of the data to be exchanged in the source database, so as to facilitate subsequent viewing of the actual situation of the backup table.

[0076] If the source database is not a distributed database, create a non-partitioned and index-free backup table.

[0077] S30: According to the general data type, copy the data to be exchanged to the standby table;

[0078] Then, according to the common data type, the data to be exchanged is copied to the standby table.

[0079] As an implementation manner, the step of copying the data to be exchanged to the standby table according to the general data type includes:

[0080] The Sqoop tool is used to read the data to be exchanged in the source database in sequence row by row, and each read row is loaded into a corresponding row of the standby table.

[0081] Sqoop is a tool for efficiently transferring data between large databases and traditional databases. Using the Sqoop tool, first connect to the source database, import the cleaned and type-converted data to be exchanged into the Sqoop tool, and then use the Sqoop tool to import the converted data into the backup table.

[0082] In the embodiment of the present invention, the data to be exchanged is first copied to the standby table. Since the loading process does not affect external services and since the standby table does not need to check indexes and other actions, the data loading efficiency can be improved.

[0083] S40: performing partition exchange on the standby table and the target table to load data in the standby table into the target table.

[0084] Finally, partitions of the standby table and the target table are exchanged to load the data in the standby table into the target table.

[0085] In some embodiments, Figure 4 As shown, the partition exchange of the standby table and the target table comprises the following steps:

[0086] S41: Determine a partition operation instruction according to the type of the target database;

[0087] S42: performing a partition operation on the target table and the backup table according to the partition operation instruction;

[0088] S43: Verify whether the partition operation is successful, and delete the backup table if successful.

[0089] First, clarify the partitioning method of the target database, such as partitioning by time, range, or hash, and understand the characteristics and data range of each partition to facilitate subsequent exchange operations; determine the backup table and target table to be exchanged. Since the data to be exchanged has been converted into a common data type that conforms to the target database in advance, it can be ensured that the data structure and partitioning method of the object to be exchanged are compatible with the source partition.

[0090] Use the syntax specific to the target database to perform partition exchange operations. Different database management systems have different syntax and commands to implement partition exchange. For example, in Oracle databases, you can use statements such as ALTER TABLE...EXCHANGEPARTITION to perform partition exchange.

[0091] After performing the exchange operation, you need to verify whether the exchange is successful. You can query the source table and target object in the source database to check whether the data is exchanged correctly. For example, you can use the following query statement to verify the exchange result: If a temporary table is used as the exchange object, after the exchange is completed, you can delete the temporary table as needed to release resources.

[0092] It can be seen that in the above scheme, for complex loan entities under the loan business, the data to be exchanged in the data table of the source database is first mapped to the general data type that conforms to the target database; then, according to the structure of the target table in the target database, a backup table with the same structure as the target table is established; then, according to the general data type, the data to be exchanged is copied to the backup table; finally, the partitions of the backup table and the target table are exchanged to load the data in the backup table into the target table.

[0093] In the embodiment of the present invention, the data to be exchanged is first copied to the spare table. Since the loading process does not affect the external services and since the spare table does not need to check the index and other actions, the data loading efficiency can be improved. In addition, the application target database can be switched in seconds, which can solve the pain points caused by batch data exchange, save labor costs, improve the platform's batch scheduling function, and at the same time ensure the normal operation of the system and enable business operations.

[0094] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0095] In one embodiment, a device for online data exchange of heterogeneous data sources is provided, and the device for online data exchange of heterogeneous data sources corresponds one-to-one to the method for online data exchange of heterogeneous data sources in the above embodiment. Figure 5 As shown, the heterogeneous data source online data exchange device includes a mapping module 501, a new creation module 502, a backup module 503 and an exchange module 504, wherein:

[0096] A mapping module 501 is used to map the data to be exchanged in the data table in the source database into a common data type that conforms to the target database;

[0097] A new creation module 502, used to create a backup table according to the structure of the target table in the target database, wherein the structure of the backup table is the same as that of the target table;

[0098] A backup module 503, used for copying the data to be exchanged to the standby table according to the general data type;

[0099] The exchange module 504 is used to perform partition exchange on the standby table and the target table to load the data in the standby table into the target table.

[0100] Among them, this embodiment is a system embodiment corresponding to the above method, and its specific implementation process is the same as the above method embodiment. For details, please refer to the above method embodiment, and this system embodiment will not go into details.

[0101] As an example, the mapping module 501 is specifically used for:

[0102] Determine the type of the target database, if the target database is a relational database, obtain the common data type of the target table, the common data type includes a primary key, a field and a data type;

[0103] Determine whether the data to be exchanged has a single primary key; if the data to be exchanged has multiple primary keys, establish a mapping relationship between the row key of the target table and the multiple primary keys;

[0104] The multiple primary keys of the data to be exchanged are used as row keys of the target table, and the data to be exchanged are mapped to the general data type.

[0105] As an example, the new module 502 is specifically used for:

[0106] If the source database is a distributed database, a non-partitioned backup table is established, and the backup table corresponds to a distribution key, and the distribution key is used to indicate the location and distribution mode of the data to be exchanged in the source database;

[0107] Otherwise, create a non-partitioned, index-free backup table.

[0108] As an example, the backup module 503 is specifically used for:

[0109] The Sqoop tool is used to read the data to be exchanged in the source database in sequence, and the data read each time is loaded into the corresponding row of the standby table.

[0110] As an example, the switching module 504 is specifically configured to:

[0111] Determine a partition operation instruction according to the type of the target database;

[0112] According to the partition operation instruction, perform a partition operation on the target table and the backup table;

[0113] Verify whether the partition operation is successful, and delete the backup table after success.

[0114] As an example, the source database is a large database.

[0115] As an example, a cleaning module is further included, and the cleaning module is used to:

[0116] The data to be exchanged is cleaned to remove duplicate data and delete blank data.

[0117] The present invention provides an online data exchange device for heterogeneous data sources. When data exchange is required between a source database and a target database, firstly, the data to be exchanged in the data table in the source database is mapped to a general data type that conforms to the target database; then, according to the structure of the target table in the target database, a backup table with the same structure as the target table is established; then, according to the general data type, the data to be exchanged is copied to the backup table; finally, the backup table and the target table are partition-exchanged to realize loading the data in the backup table into the target table.

[0118] In the embodiment of the present invention, the data to be exchanged is first copied to the spare table. Since the loading process does not affect the external services and since the spare table does not need to check the index and other actions, the data loading efficiency can be improved. In addition, the application target database can be switched in seconds, which can solve the pain points caused by batch data exchange, save labor costs, improve the platform's batch scheduling function, and at the same time ensure the normal operation of the system and enable business operations.

[0119] For the specific definition of the online data exchange device for heterogeneous data sources, please refer to the definition of the online data exchange method for heterogeneous data sources above, which will not be repeated here. Each module in the above-mentioned online data exchange device for heterogeneous data sources can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0120] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of an intelligent question-answering processing method based on artificial intelligence.

[0121] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of the client side of an intelligent question-answering processing method based on artificial intelligence

[0122] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0123] Map the data to be exchanged in the data table in the source database to the common data type that conforms to the target database;

[0124] According to the structure of the target table in the target database, a new backup table is created, wherein the structure of the backup table is the same as that of the target table;

[0125] According to the general data type, copying the data to be exchanged to the standby table;

[0126] Partition exchange is performed on the standby table and the target table to load data in the standby table into the target table.

[0127] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0128] Map the data to be exchanged in the data table in the source database to the common data type that conforms to the target database;

[0129] According to the structure of the target table in the target database, a new backup table is created, wherein the structure of the backup table is the same as that of the target table;

[0130] According to the general data type, copying the data to be exchanged to the standby table;

[0131] Partition exchange is performed on the standby table and the target table to load data in the standby table into the target table.

[0132] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0134] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0135] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for online data exchange of heterogeneous data sources, characterized in that: include: Map the data to be exchanged in the data table in the source database to the common data type that conforms to the target database; According to the structure of the target table in the target database, a new backup table is created, wherein the structure of the backup table is the same as that of the target table; According to the general data type, copying the data to be exchanged to the standby table; Partition exchange is performed on the standby table and the target table to load data in the standby table into the target table.

2. The method for online data exchange of heterogeneous data sources according to claim 1, characterized in that: The step of mapping the data to be exchanged in the data table in the source database to a common data type that conforms to the target database includes: Determine the type of the target database, if the target database is a relational database, obtain the common data type of the target table, the common data type includes a primary key, a field and a data type; Determine whether the data to be exchanged has a single primary key; if the data to be exchanged has multiple primary keys, establish a mapping relationship between the row key of the target table and the multiple primary keys; The multiple primary keys of the data to be exchanged are used as row keys of the target table, and the data to be exchanged are mapped to the general data type.

3. The method for online data exchange of heterogeneous data sources according to claim 1, characterized in that: The backup table is a bak table, and the steps of creating a new backup table according to the structure of the target table in the target database include: If the source database is a distributed database, a non-partitioned backup table is established, and the backup table corresponds to a distribution key, and the distribution key is used to indicate the location and distribution mode of the data to be exchanged in the source database; Otherwise, create a non-partitioned, index-free backup table.

4. The method for online data exchange of heterogeneous data sources according to claim 1, characterized in that: The step of copying the data to be exchanged to the standby table according to the general data type comprises: The Sqoop tool is used to read the data to be exchanged in the source database in sequence, and the data read each time is loaded into the corresponding row of the standby table.

5. The method for online data exchange of heterogeneous data sources according to claim 1, characterized in that: The step of performing partition exchange on the standby table and the target table comprises: Determine a partition operation instruction according to the type of the target database; According to the partition operation instruction, perform a partition operation on the target table and the backup table; Verify whether the partition operation is successful, and delete the backup table after success.

6. The method for online data exchange of heterogeneous data sources according to any one of claims 1 to 5, characterized in that: The source database is a large database.

7. The method for online data exchange of heterogeneous data sources according to any one of claims 1 to 5, characterized in that: Before the step of mapping the data to be exchanged in the data table of the source database into a common data type that conforms to the target database, the step further includes: The data to be exchanged is cleaned to remove duplicate data and delete blank data.

8. An online data exchange device for heterogeneous data sources, characterized in that: include: A mapping module, used to map the data to be exchanged in the data table in the source database into a common data type that conforms to the target database; A new creation module, used for creating a new backup table according to the structure of the target table in the target database, wherein the structure of the backup table is the same as that of the target table; A backup module, used for copying the data to be exchanged to the standby table according to the general data type; The exchange module is used to perform partition exchange on the standby table and the target table to load the data in the standby table into the target table.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method for online data exchange of heterogeneous data sources as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for online data exchange of heterogeneous data sources as claimed in any one of claims 1 to 7 are implemented.

Images