Data processing method, device, equipment and computer readable storage medium

Abstract

A data processing method, device, equipment and computer-readable storage medium, relating to the technical field of database data processing, including creating a remote connection access between a destination database and a source database through a preset callback function and a native interface of a source database, wherein the destination database includes a temporary data table, and the temporary data table has the same structure as a data table to be migrated in the source database; reading the data to be migrated in the data table to be migrated based on the temporary data table; converting the read data to be migrated according to a preset conversion function to obtain target data, wherein the target data is adapted to the destination database; and writing the target data into a target data table in the destination database to realize data migration. The present application can effectively realize the direct migration of data between different types of databases.

Description

Technical Field

[0001] The present application relates to the technical field of database data processing, and in particular to a data processing method, device, equipment and computer-readable storage medium. Background Art

[0002] Under the joint efforts of the three driving forces of database technology development, digital transformation applications, and information innovation, databases have gradually evolved from a single architecture supporting multiple types of applications to multiple architectures supporting multiple types of applications. However, these architectures are not replacement relationships, but rather coexistence and common development relationships, so the diversity of databases has become inevitable; in addition, the infrastructure that supports database operations has also evolved from the traditional three-tier IT architecture to various cloud architectures, bringing the database industry into the era of diversified databases.

[0003] When a system needs to support multiple databases, there is often a need to store the same data, so there will inevitably be compatibility issues. However, commonly used database backup and recovery tools are only applicable to the import and export of databases of the same type, so the exported data cannot be directly imported into other different types of databases. It can be seen that how to achieve data migration between different types of databases is an urgent problem that needs to be solved. Summary of the invention

[0004] The present application provides a data processing method, apparatus, device and computer-readable storage medium, which can solve the technical problem in the prior art that data cannot be directly migrated between different types of databases.

[0005] In a first aspect, an embodiment of the present application provides a data processing method, the data processing method comprising:

[0006] Creating a remote connection access between a sink database and a source database through a preset callback function and a native interface of the source database, wherein the sink database includes a temporary data table having the same structure as a data table to be migrated in the source database;

[0007] Reading the data to be migrated in the data table to be migrated based on the temporary data table;

[0008] Converting the read data to be migrated according to a preset conversion function to obtain target data, wherein the target data is adapted to the destination database;

[0009] The target data is written into a target data table in the host database to achieve data migration.

[0010] In combination with the first aspect, in one implementation, the callback function includes a server callback function, a user group callback function, and an external table callback function, and the remote connection access between the host database and the source database is created through the preset callback function and the native interface of the source database, including:

[0011] Calling the native interface through the server callback function to remotely connect the sink database to the source database;

[0012] Obtain user information of the source database based on the user group callback function, so that the sink database can remotely access the source database;

[0013] The table information of the data table to be migrated in the source database is obtained according to the external table callback function, so as to construct the temporary data table in the destination database based on the table information.

[0014] In combination with the first aspect, in one implementation, reading the data to be migrated in the data table to be migrated based on the temporary data table includes:

[0015] The external table callback function is called based on the temporary data table, so as to read the data to be migrated from the data table to be migrated through the external table callback function.

[0016] In combination with the first aspect, in one implementation, the method further includes:

[0017] Adding a column of source-end fields to the to-be-migrated data table, and adding a column of sink-end fields to the target data table;

[0018] For each row in the data table to be migrated, perform hash calculation on the non-newly added fields and write the calculated source-side hash value into the corresponding source-side field;

[0019] For each row in the target data table, perform hash calculation on the non-newly added fields, and write the calculated sink hash value into the corresponding sink field;

[0020] The difference data between the to-be-migrated data table and the target data table is determined according to the consistency between the source-end hash value and the sink-end hash value.

[0021] In combination with the first aspect, in one implementation, determining the difference data between the to-be-migrated data table and the target data table according to the consistency between the source-end hash value and the sink-end hash value includes:

[0022] Comparing all source hash values ​​and all sink hash values ​​to determine a first source hash value and a first sink hash value, wherein the first source hash value is different from all sink hash values ​​in the target data table, and the first sink hash value is different from all source hash values ​​in the source data table;

[0023] Deduplication processing is performed on each column of data in the row where the first source-end hash value is located and each column of data in the row where the first sink-end hash value is located to obtain difference data.

[0024] In combination with the first aspect, in one implementation, after the step of comparing all source-end hash values ​​with all sink-end hash values, the method further includes:

[0025] If each source hash value has a corresponding sink hash value and each sink hash value has a corresponding source hash value, it is determined that there is no difference data between the to-be-migrated data table and the target data table.

[0026] In combination with the first aspect, in one implementation, after the step of determining the difference data between the to-be-migrated data table and the target data table according to the consistency between the source-end hash value and the sink-end hash value, the method further includes:

[0027] The source end field, the sink end field and the temporary data table are deleted.

[0028] In a second aspect, an embodiment of the present application provides a data processing device, the data processing device comprising:

[0029] A remote control module, which is used to create a remote connection access between a sink database and a source database through a preset callback function and a native interface of the source database, wherein the sink database includes a temporary data table having the same structure as the data table to be migrated in the source database;

[0030] A data reading module, configured to read the data to be migrated in the data table to be migrated based on the temporary data table;

[0031] A data conversion module, which is used to convert the read data to be migrated according to a preset conversion function to obtain target data, wherein the target data is adapted to the destination database;

[0032] A data migration module is used to write the target data into a target data table in the host database to achieve data migration.

[0033] In conjunction with the second aspect, in one implementation, the callback function includes a server callback function, a user group callback function, and an external table callback function, and the remote control module is specifically used to:

[0034] Calling the native interface through the server callback function to remotely connect the sink database to the source database;

[0035] Obtain user information of the source database based on the user group callback function, so that the sink database can remotely access the source database;

[0036] The table information of the data table to be migrated in the source database is obtained according to the external table callback function, so as to construct the temporary data table in the destination database based on the table information.

[0037] In conjunction with the second aspect, in one implementation, the data reading module is specifically used to:

[0038] The external table callback function is called based on the temporary data table, so as to read the data to be migrated from the data table to be migrated through the external table callback function.

[0039] In conjunction with the second aspect, in one implementation, the data processing device further includes a data comparison module, which is used to:

[0040] Adding a column of source-end fields to the to-be-migrated data table, and adding a column of sink-end fields to the target data table;

[0041] For each row in the data table to be migrated, perform hash calculation on the non-newly added fields and write the calculated source-side hash value into the corresponding source-side field;

[0042] For each row in the target data table, perform hash calculation on the non-newly added fields, and write the calculated sink hash value into the corresponding sink field;

[0043] The difference data between the to-be-migrated data table and the target data table is determined according to the consistency between the source-end hash value and the sink-end hash value.

[0044] In conjunction with the second aspect, in one implementation, the data comparison module is specifically used to:

[0045] Comparing all source hash values ​​and all sink hash values ​​to determine a first source hash value and a first sink hash value, wherein the first source hash value is different from all sink hash values ​​in the target data table, and the first sink hash value is different from all source hash values ​​in the source data table;

[0046] Deduplication processing is performed on each column of data in the row where the first source-end hash value is located and each column of data in the row where the first sink-end hash value is located to obtain difference data.

[0047] In conjunction with the second aspect, in one implementation, the data comparison module is further used to:

[0048] If each source hash value has a corresponding sink hash value and each sink hash value has a corresponding source hash value, it is determined that there is no difference data between the to-be-migrated data table and the target data table.

[0049] In combination with the second aspect, in one implementation, the data processing device further includes a table processing module, which is used to: delete the source field, the sink field and the temporary data table.

[0050] In a third aspect, an embodiment of the present application provides a data processing device, comprising a processor, a memory, and a data processing program stored in the memory and executable by the processor, wherein when the data processing program is executed by the processor, the steps of the data processing method as described above are implemented.

[0051] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a data processing program is stored, wherein when the data processing program is executed by a processor, the steps of the aforementioned data processing method are implemented.

[0052] The beneficial effects brought by the technical solution provided by the embodiment of the present application include at least:

[0053] A "remote data connector" is constructed through a callback function to call the native interface of the source database based on the "remote data connector" tool to create a remote connection access between the host database and the source database; wherein, the "remote data connector" will create a temporary data table with the same structure as the data table to be migrated in the source database in the host database, so as to use the temporary data table as a link to access the data to be migrated in the data table to be migrated; then the read data to be migrated is converted according to the conversion function to solve the compatibility problem between different types of databases, so that the converted target data can be adapted to the host database; finally, the target data is written into the target data table in the host database to realize data migration. It can be seen that the incompatibility problems caused by different data definitions, constraints, etc. between different types of databases can be effectively solved through this application, so as to realize the direct migration of data between different types of databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 This is a flow chart of an embodiment of the data processing method of the present application;

[0055] Figure 2 This is a schematic diagram of data interaction between source and sink databases involved in the embodiment of the present application;

[0056] Figure 3 A schematic diagram of data mapping between a data table to be migrated and a temporary data table involved in an embodiment of the present application;

[0057] Figure 4 A schematic diagram of data mapping between different data tables involved in the embodiment of the present application;

[0058] Figure 5 This is a schematic diagram of the newly added fields involved in the embodiment of the present application;

[0059] Figure 6 This is a schematic diagram of the difference data involved in the embodiment of the present application;

[0060] Figure 7 This is a schematic diagram of the hardware structure of the data processing device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0063] In a first aspect, an embodiment of the present application provides a data processing method.

[0064] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of an embodiment of the data processing method of this application. Figure 1 As shown, the data processing method includes:

[0065] Step S10: creating a remote connection access between the sink database and the source database through a preset callback function and a native interface of the source database, wherein the sink database includes a temporary data table having the same structure as the data table to be migrated in the source database.

[0066] Exemplarily, it should be understood that the source database is a database where data to be migrated exists, and the destination database is a database used to store the data to be migrated in the source database. For example, database A includes data to be migrated and needs to migrate the data to be migrated to database B, then database A is the source database, and database B is the destination database; the callback function refers to a function passed as a parameter. In this embodiment, the data interaction between the destination database and the source database is realized through the callback function; the native interface of the source database refers to the interface directly provided by the database system for external applications to interact and operate, that is, the native interface provides a bridge to connect to the source database, which exposes the interface of IP, port, user, password, and table information to the outside for client to call and access.

[0067] In this embodiment, a "remote data connector" is constructed at the sink end, so as to realize data migration between the sink end database and the source end database through the tool of the "remote data connector". Specifically, first, the official native interface lib_A.so of the source database A is pre-downloaded in the host database B, so that the native interface lib_A.so is used as a bridge for data interaction between the host database B and the source database A; secondly, a callback function for obtaining the source database A and its data table tA to be migrated and the data to be migrated is constructed, that is, the source database A and its data table tA to be migrated can be passed in through the callback function and the native interface lib_A.so to achieve remote connection access between the host database B and the source database A, and then a temporary data table t_tmp with the same structure as the data table tA to be migrated in the source database A can be created in the host database B based on the relevant information; wherein the temporary data table t_tmp can be used as a link to access and read the data to be migrated in the data table tA to be migrated, that is, the data in the data table tA to be migrated under the source database A is mapped to the temporary data table t_tmp under the host database B.

[0068] Further, in one embodiment, the callback function includes a server callback function, a user group callback function and an external table callback function, and the remote connection access between the host database and the source database is created through the preset callback function and the native interface of the source database, including:

[0069] Calling the native interface through the server callback function to remotely connect the sink database to the source database;

[0070] Obtain user information of the source database based on the user group callback function, so that the sink database can remotely access the source database;

[0071] The table information of the data table to be migrated in the source database is obtained according to the external table callback function, so as to construct the temporary data table in the destination database based on the table information.

[0072] For example, see Figure 2 As shown, this embodiment will create a custom server callback function SERVER, which is used to remotely connect to the source database A through lib_A.so; specifically, by passing the host information of the source database A including the IP, port information, host name, etc. and the instance name of the database service to the SERVER function, the SERVER function will call the database connection interface in lib_A.so to remotely connect to the source database A, so as to realize the remote connection between the host database B and the source database A.

[0073] Secondly, a custom user group callback function USER will be created. Its function is to remotely access the table under the source database A through the SERVER function. Specifically, by passing user information such as the user name user and password passward of the source database A to the USER function, the USER function will call back the user information to the SERVER to access the source database A, thereby realizing remote access between the host database B and the source database A.

[0074] Then, create a custom external table callback function TABLE, which maps the data of the to-be-migrated data table tA under the source database A to the table t_tmp under the destination database B through the SERVER function and the USER function. Specifically, by passing the table name table, mode schema, table definition and other table information of the to-be-migrated data table tA to the TABLE function, the data of the to-be-migrated data table tA can be read through the TABLE function.

[0075] After obtaining the table information, a temporary data table t_tmp having the same structure as the data table tA to be migrated is created in the host database B according to the table information, so as to link to the data table tA to be migrated through the temporary data table t_tmp, thereby realizing the reading of the data in the data table tA to be migrated. It should be noted that the field names of the temporary data table t_tmp are consistent with those of the data table tA to be migrated, and the field types are all text types except integer and time types.

[0076] Specifically, assuming that the source database A is MySQL and the host database B is PostgreSQL, the native interface is libmysqlclient.so, and the libmysqlclient.so interface will be downloaded in the host database B; then the callback function SERVER is called in the host database B, and the upper-level application in the system will pass the IP and port information of the source database A to the callback function SERVER to connect to the source database A; then the callback function USER is called, and the upper-level application will pass the user name and password of the source database A to the callback function USER to access the data table tA to be migrated under the source database A; finally, the callback function TABLE is called, and the upper-level application will pass the table name and other table information of the data table tA to be migrated of the source database A to the callback function TABLE to read the data of the data table tA to be migrated under the source database A.

[0077] Step S20: reading the data to be migrated in the data table to be migrated based on the temporary data table; wherein the external table callback function is called based on the temporary data table to read the data to be migrated from the data table to be migrated through the external table callback function.

[0078] For example, in this embodiment, since the temporary data table t_tmp and the to-be-migrated data table tA have exactly the same structure, the temporary data table t_tmp can be used as a link to access and read the to-be-migrated data in the to-be-migrated data table tA, so as to map the data in the to-be-migrated data table tA to the temporary data table t_tmp. Figure 3 As shown, by clicking on a cell of the temporary data table t_tmp, you can jump to the corresponding cell in the data table tA to be migrated to read the data to be migrated under the cell; for example, by clicking on the cell X in the 1st row and 5th column of the temporary data table t_tmp, you will jump to access the cell Y in the 1st row and 5th column of the data table tA to be migrated, and then you can read the data to be migrated under cell Y.

[0079] Step S30: converting the read data to be migrated according to a preset conversion function to obtain target data, wherein the target data is adapted to the destination database.

[0080] Exemplarily, it should be understood that data definitions, constraints, etc. between different types of databases are usually different, resulting in incompatibility issues; for example, with respect to line breaks in a MySQL database, data will be wrapped when exported via a file, and a line separator can be specified during import, so that the wrapped data can be stored in the table as is, but these wrapped data cannot be imported into a PostgreSQL-type database, because the PostgreSQL database has no concept of line separators, i.e., a line break represents a new row of data, i.e., the line break in PostgreSQL will be explicitly converted to a \n character, and PostgreSQL will recognize the \n character and convert it to a line break when importing; similarly, the \n in PostgreSQL cannot be correctly converted to a line break in MySQL, resulting in data inconsistency.

[0081] In order to solve the incompatibility problem, this embodiment will create an ETL (Extract, Transform, Load) conversion function on the sink end to achieve compatibility conversion between the source database A and the sink database B, that is, convert the read data to be migrated so that the data definition, constraint conditions, format, etc. of the converted target data can match the sink database B; for example, the time type of the MySQL database supports 0000-00-00 00:00:00, but the PostgreSQL database only supports 1970-01-01 00:00:00 at the minimum, so it is necessary to use the ETL conversion function to convert the time; for another example, when migrating from Windows database A to Linux database B, there will be a difference in the path drive letter, so it is necessary to use the ETL conversion function to convert "D: / " into the " / opt" path writing. It should be noted that the specific construction and implementation principle of the ETL conversion function and how to realize data conversion through the ETL conversion function are common knowledge in the field and will not be repeated here.

[0082] Step S40: writing the target data into the target data table in the destination database to achieve data migration.

[0083] For example, see Figure 4 As shown, after the data conversion is realized through the ETL conversion function, the converted target data can be directly written into the target data table tB in the host database B, thereby realizing data migration. It can be seen that the "remote data connector" tool created in this application can effectively solve the incompatibility problems caused by different data definitions, constraints, etc. between different types of databases, so as to realize the direct migration of data between different types of databases.

[0084] Furthermore, in one embodiment, the method further includes:

[0085] Adding a column of source-end fields to the to-be-migrated data table, and adding a column of sink-end fields to the target data table;

[0086] For each row in the data table to be migrated, perform hash calculation on the non-newly added fields and write the calculated source-side hash value into the corresponding source-side field;

[0087] For each row in the target data table, perform hash calculation on the non-newly added fields, and write the calculated sink hash value into the corresponding sink field;

[0088] The difference data between the to-be-migrated data table and the target data table is determined according to the consistency between the source-end hash value and the sink-end hash value.

[0089] For example, see Figure 5 As shown, a new column of source-side field concat_col (i.e., T field) will be added to the data table to be migrated tA, and the field type is text; wherein, for each row in the data table to be migrated tA, a concatenation function (such as concat function) is used to concatenate and hash all fields in the row except the source-side field to obtain the source-side hash value hash corresponding to the row, and the source-side hash value hash is written into the corresponding position of the concat_col column. Similarly, a new column of destination-side field concat_col is added to the target data table tB, and the field type is also text; wherein, for each row in the target data table tB, a concat function is also used to concatenate and hash all fields in the row except the destination field to obtain the destination-side hash value hash corresponding to the row, and the destination-side hash value hash is written into the corresponding position of the concat_col column.

[0090] It should be understood that if the source hash value is the same as the destination hash value, it means that the data in the row where the source hash value is located is consistent with the data in the row where the destination hash value is located; therefore, it can be determined whether there is difference data between the data table tA to be migrated and the target data table tB based on the consistency between the source hash value and the destination hash value, that is, whether there is a source hash value in the data table tA to be migrated that cannot be adapted to the target data table tB and whether there is a destination hash value in the target data table tB that cannot be adapted to the data table tA to be migrated; if there is at least one source hash value that cannot be adapted in the data table tB to be migrated or at least one destination hash value that cannot be adapted in the target data table tB, it is determined that there is difference data, that is, an error occurs before and after data migration. It can be seen that the "remote data connector" can accurately verify the correctness of data.

[0091] Further, in one embodiment, the determining the difference data between the to-be-migrated data table and the target data table according to the consistency between the source-end hash value and the sink-end hash value includes:

[0092] Comparing all source hash values ​​and all sink hash values ​​to determine a first source hash value and a first sink hash value, wherein the first source hash value is different from all sink hash values ​​in the target data table, and the first sink hash value is different from all source hash values ​​in the source data table;

[0093] Deduplication processing is performed on each column of data in the row where the first source-end hash value is located and each column of data in the row where the first sink-end hash value is located to obtain difference data.

[0094] For example, in this embodiment, data verification between the to-be-migrated data table tA and the target data table tB is implemented in the host database, that is, only the data in the to-be-migrated data table tA needs to be obtained to be compared with the data in the target data table tB. This is more efficient than the traditional method of moving both sides of the data into the memory for comparison, and can greatly reduce memory overhead. For details, see Figure 6 As shown, first, the source hash value in the data table tA to be migrated is read through the temporary data table t_tmp, and it is determined whether there is a destination hash value in the target data table tB that is the same as the source hash value read. If so, it indicates that the row data where the source hash value is located exists in both the temporary data table t_tmp (that is, the data table tA to be migrated) and the target data table tB; if not, it indicates that the row data where the source hash value is located only exists in the temporary data table t_tmp, while the row data where the source hash value is located does not exist in the target data table tB. Therefore, it is determined that the source hash value is the first source hash value, and the row data where the first source hash value is located can be stored as t_tmp_line; similarly, the row data where the first destination hash value that only exists in the target data table tB and is different from all source hash values ​​in the temporary data table t_tmp can be stored as t_line; based on this, all the first source hash values ​​in the temporary data table t_tmp and all the first destination hash values ​​in the target data table tB are found.

[0095] It should be understood that the consistency check between the source end hash value and the host end hash value can be achieved through an association algorithm, that is, the row in the temporary data table t_tmp and not in the target data table tB is taken out through the association algorithm, and recorded as t_tmp_line; similarly, the row in the target data table tB and not in the temporary data table t_tmp is taken out from the target data table tB and recorded as t_line. Among them, the association calculation refers to taking the Cartesian product of the target data table tB and the data table tA to be migrated, and then taking the target data table tB as the base table, removing the records in the Cartesian product that are divided into NULL in the target data table tB, that is, removing the rows in the data table tA to be migrated that are not matched with the records of the target data table tB; at the same time, taking the data table tA to be migrated as the base table, removing the records in the Cartesian product that are divided into NULL in the data table tA to be migrated, that is, removing the rows in the target data table tB that are not matched with the records of the data table tA to be migrated.

[0096] After the comparison algorithm is executed, this embodiment will only write a small amount of different data into the memory for deduplication filtering; for details, see Figure 6 As shown, the column values ​​in t_tmp_line and t_line are converted into a json data structure (i.e., field name: value). That is, for each row of t_tmp_line, the value of each column in the t_tmp_line row is converted into a key-value pair, saved in the form of json, recorded as t_tmp_json; similarly, for each row of t_line, the value of each column in the t_line row is converted into a key-value pair, saved in the form of json, recorded as t_json.

[0097] It should be understood that for a row of data with different source-end hash values ​​and sink-end hash values, there may be some columns whose values ​​are the same, that is, the data in this column is correct before and after migration, so it is necessary to deduplicate this type of data; for example, for t_tmp_line and t_line with row number 3, assuming that the data in the 5th column of t_tmp_line is the same as the data in the 5th column of t_line, and the other column data are different, then the data in the 5th column of t_tmp_line and t_line need to be removed, and only the other different data are retained. Therefore, in this embodiment, t_tmp_json and t_json can be first converted into a set, and then the set can be converted into a set data structure (that is, a series of unordered data sets without duplicate values) to obtain t_tmp_set and t_set, and then t_tmp_set and t_set are filtered for duplicate values, so as to complete the determination of the difference data.

[0098] See also Figure 6As shown, this embodiment can also create a result table for storing difference data in the destination database B, and the fields of the table are: table name, source difference data, and destination difference data, so as to store the difference sets of t_tmp_set and t_set in the result table, that is, the table displays inconsistent data; and then display the result table in a visual form, that is, write the difference data into the final display table, and each row only displays the different data key-value pairs of the source and destination of a table to improve friendliness. It should be noted that if the data on both sides are completely consistent, t_tmp_set and t_set are empty, and there is no need to write them into the result table.

[0099] Furthermore, in one embodiment, after the step of comparing all source-end hash values ​​with all sink-end hash values, the method further includes:

[0100] If each source hash value has a corresponding sink hash value and each sink hash value has a corresponding source hash value, it is determined that there is no difference data between the to-be-migrated data table and the target data table.

[0101] Exemplarily, in this embodiment, if each source hash value can find the corresponding sink hash value in the target data table tB and each sink hash value can find the corresponding source hash value in the data table to be migrated tA, it means that the data in the data table to be migrated tA has been correctly written into the target data table tB and no additional error data has been written into the target data table tB. It can be determined that there is no difference data between the data table to be migrated tA and the target data table tB, that is, no error occurred before and after the data migration. It can be seen that the correctness of data before and after migration can be effectively guaranteed by the "remote data connector".

[0102] Furthermore, in one embodiment, after the step of determining the difference data between the to-be-migrated data table and the target data table according to the consistency between the source-end hash value and the sink-end hash value, the step further includes:

[0103] The source end field, the sink end field and the temporary data table are deleted.

[0104] Exemplarily, in this embodiment, after completing the comparison between the data table to be migrated tA and the target data table tB, in order to further reduce memory usage, the newly added source fields in the data table to be migrated tA and the newly added host fields in the target data table tB are deleted, and the temporary data table t_tmp created in the host database B is deleted.

[0105] In summary, this embodiment remotely accesses the data of the source database and performs data conversion and migration through the "remote data connector" created on the host database side, which can effectively solve the incompatibility problems caused by different data definitions and constraints between different databases, and identifies differential data through comparison algorithms to verify whether the data meets the consistency requirements, thereby ensuring the accuracy before and after data migration.

[0106] It should be noted that, in this embodiment, the two methods of data migration and data verification can be independent of each other, that is, data verification may not be performed during data migration, and data verification does not necessarily require data migration, and two existing database data can be directly compared.

[0107] In a second aspect, an embodiment of the present application also provides a data processing device.

[0108] In one embodiment, the data processing device includes:

[0109] A remote control module, which is used to create a remote connection access between a sink database and a source database through a preset callback function and a native interface of the source database, wherein the sink database includes a temporary data table having the same structure as the data table to be migrated in the source database;

[0110] A data reading module, configured to read the data to be migrated in the data table to be migrated based on the temporary data table;

[0111] A data conversion module, which is used to convert the read data to be migrated according to a preset conversion function to obtain target data, wherein the target data is adapted to the destination database;

[0112] A data migration module is used to write the target data into a target data table in the host database to achieve data migration.

[0113] Further, in one embodiment, the callback function includes a server callback function, a user group callback function and an external table callback function, and the remote control module is specifically used for:

[0114] Calling the native interface through the server callback function to remotely connect the sink database to the source database;

[0115] Obtain user information of the source database based on the user group callback function, so that the sink database can remotely access the source database;

[0116] The table information of the data table to be migrated in the source database is obtained according to the external table callback function, so as to construct the temporary data table in the destination database based on the table information.

[0117] Furthermore, in one embodiment, the data reading module is specifically used for:

[0118] The external table callback function is called based on the temporary data table, so as to read the data to be migrated from the data table to be migrated through the external table callback function.

[0119] Furthermore, in one embodiment, the data processing device further includes a data comparison module, which is used to:

[0120] Adding a column of source-end fields to the to-be-migrated data table, and adding a column of sink-end fields to the target data table;

[0121] For each row in the data table to be migrated, perform hash calculation on the non-newly added fields and write the calculated source-side hash value into the corresponding source-side field;

[0122] For each row in the target data table, perform hash calculation on the non-newly added fields, and write the calculated sink hash value into the corresponding sink field;

[0123] The difference data between the to-be-migrated data table and the target data table is determined according to the consistency between the source-end hash value and the sink-end hash value.

[0124] Furthermore, in one embodiment, the data comparison module is specifically used for:

[0125] Comparing all source hash values ​​and all sink hash values ​​to determine a first source hash value and a first sink hash value, wherein the first source hash value is different from all sink hash values ​​in the target data table, and the first sink hash value is different from all source hash values ​​in the source data table;

[0126] Deduplication processing is performed on each column of data in the row where the first source-end hash value is located and each column of data in the row where the first sink-end hash value is located to obtain difference data.

[0127] Furthermore, in one embodiment, the data comparison module is further used to:

[0128] If each source hash value has a corresponding sink hash value and each sink hash value has a corresponding source hash value, it is determined that there is no difference data between the to-be-migrated data table and the target data table.

[0129] Furthermore, in one embodiment, the data processing device further includes a table processing module, which is used to delete the source field, the sink field and the temporary data table.

[0130] Among them, the functional implementation of each module in the above-mentioned data processing device corresponds to each step in the above-mentioned data processing method embodiment, and its functions and implementation processes are no longer repeated here one by one.

[0131] In a third aspect, an embodiment of the present application provides a data processing device, which may be a personal computer (PC), a laptop computer, a server, or other device with data processing capabilities.

[0132] Reference Figure 7 , Figure 7 Schematic diagram of the hardware structure of the data processing device involved in the embodiment of the present application. In the embodiment of the present application, the data processing device may include a processor, a memory, a communication interface and a communication bus.

[0133] The communication bus may be of any type and is used to interconnect the processor, the memory, and the communication interface.

[0134] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces, which are used to interconnect devices within a data processing device, as well as interfaces used to interconnect a data processing device with other devices (such as other computing devices or user devices). Physical interfaces can be Ethernet interfaces, optical fiber interfaces, ATM (Asynchronous Transfer Mode) interfaces, etc.; user devices can be display screens (Display), keyboards (Keyboard), etc.

[0135] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0136] The processor may be a general-purpose processor, and the general-purpose processor may call the data processing program stored in the memory and execute the data processing method provided in the embodiment of the present application. For example, the general-purpose processor may be a central processing unit (CPU). The method executed when the data processing program is called may refer to the various embodiments of the data processing method of the present application, and will not be repeated here.

[0137] Those skilled in the art will understand that Figure 7 The hardware structure shown in the figure does not constitute a limitation on the present application, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0138] In a fourth aspect, an embodiment of the present application also provides a computer-readable storage medium.

[0139] The readable storage medium of the present application stores a data processing program, wherein when the data processing program is executed by a processor, the steps of the data processing method as described above are implemented.

[0140] Among them, the method implemented when the data processing program is executed can refer to the various embodiments of the data processing method of the present application, and will not be repeated here.

[0141] It should be noted that the serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0142] The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit "first", "second" and "third" to different types.

[0143] In the description of the embodiments of the present application, "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a specific way.

[0144] In the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; the “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0145] In some processes described in the embodiments of the present application, multiple operations or steps that appear in a specific order are included, but it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present application or in parallel, and the sequence number of the operation is only used to distinguish the different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0146] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD) as described above, and includes a number of instructions for a terminal device to execute the methods described in each embodiment of the present application.

[0147] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Creating a remote connection access between a sink database and a source database through a preset callback function and a native interface of the source database, wherein the sink database includes a temporary data table having the same structure as a data table to be migrated in the source database; Reading the data to be migrated in the data table to be migrated based on the temporary data table; Converting the read data to be migrated according to a preset conversion function to obtain target data, wherein the target data is adapted to the destination database; The target data is written into a target data table in the host database to achieve data migration.

2. The data processing method according to claim 1, characterized in that: The callback function includes a server callback function, a user group callback function and an external table callback function. The remote connection access between the host database and the source database is created through the preset callback function and the native interface of the source database, including: Calling the native interface through the server callback function to remotely connect the sink database to the source database; Obtain user information of the source database based on the user group callback function, so that the sink database can remotely access the source database; The table information of the data table to be migrated in the source database is obtained according to the external table callback function, so as to construct the temporary data table in the destination database based on the table information.

3. The data processing method according to claim 2, characterized in that: The reading the data to be migrated in the data table to be migrated based on the temporary data table includes: The external table callback function is called based on the temporary data table, so as to read the data to be migrated from the data table to be migrated through the external table callback function.

4. The data processing method according to claim 1, characterized in that: The method further comprises: Adding a column of source-end fields to the to-be-migrated data table, and adding a column of sink-end fields to the target data table; For each row in the data table to be migrated, perform hash calculation on the non-newly added fields and write the calculated source-side hash value into the corresponding source-side field; For each row in the target data table, perform hash calculation on the non-newly added fields, and write the calculated sink hash value into the corresponding sink field; The difference data between the to-be-migrated data table and the target data table is determined according to the consistency between the source-end hash value and the sink-end hash value.

5. The data processing method according to claim 4, characterized in that: The determining the difference data between the to-be-migrated data table and the target data table according to the consistency between the source end hash value and the sink end hash value includes: Comparing all source hash values ​​and all sink hash values ​​to determine a first source hash value and a first sink hash value, wherein the first source hash value is different from all sink hash values ​​in the target data table, and the first sink hash value is different from all source hash values ​​in the source data table; Deduplication processing is performed on each column of data in the row where the first source-end hash value is located and each column of data in the row where the first sink-end hash value is located to obtain difference data.

6. The data processing method according to claim 5, characterized in that: After the step of comparing all source-end hash values ​​with all sink-end hash values, the method further includes: If each source hash value has a corresponding sink hash value and each sink hash value has a corresponding source hash value, it is determined that there is no difference data between the to-be-migrated data table and the target data table.

7. The data processing method according to claim 4, characterized in that: After the step of determining the difference data between the to-be-migrated data table and the target data table according to the consistency between the source-end hash value and the sink-end hash value, the method further includes: The source end field, the sink end field and the temporary data table are deleted.

8. A data processing device, characterized in that: The data processing device comprises: A remote control module, which is used to create a remote connection access between a sink database and a source database through a preset callback function and a native interface of the source database, wherein the sink database includes a temporary data table having the same structure as the data table to be migrated in the source database; A data reading module, configured to read the data to be migrated in the data table to be migrated based on the temporary data table; A data conversion module, which is used to convert the read data to be migrated according to a preset conversion function to obtain target data, wherein the target data is adapted to the destination database; A data migration module is used to write the target data into a target data table in the host database to achieve data migration.

9. A data processing device, characterized in that: The data processing device comprises a processor, a memory, and a data processing program stored in the memory and executable by the processor, wherein when the data processing program is executed by the processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data processing program, wherein when the data processing program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 7 are implemented.

Images