A database data processing method, apparatus, and computer equipment

Abstract

This disclosure provides a data processing method, apparatus, and computer device for a database. The method includes: acquiring data to be migrated and determining a migration statement based on the data to be migrated; determining compatibility information of a target database for the migration statement; determining target data corresponding to the compatibility information in the data to be migrated, and migrating the target data to the target database.

Description

Technical Field

[0001] This disclosure relates to the field of database technology, and more specifically, to a database data processing method, apparatus, and computer equipment. Background Technology

[0002] The rise of the internet industry has spurred the rise and rapid development of distributed technologies. More and more industries are adopting distributed database systems for data storage, such as the financial industry, to meet the needs of online, high-frequency, multi-dimensional, and high-concurrency scenarios in the era of industrial internet. This not only solves technical bottlenecks but also broadens the depth and breadth of business services.

[0003] However, due to the diverse types of databases in distributed database systems, data migration between different databases is inevitable. However, different types of databases often have differences in data types and syntax, leading to incompatibility. This incompatibility can cause migration obstacles and failures during the migration process. Summary of the Invention

[0004] This disclosure provides at least one data processing method, apparatus, and computer device for a database.

[0005] In a first aspect, embodiments of this disclosure provide a data processing method for a database, including:

[0006] Obtain the data to be migrated, and determine the migration statement based on the data to be migrated;

[0007] Determine the compatibility information of the target database with the migration statement;

[0008] The target data corresponding to the compatibility information in the data to be migrated is determined, and the target data is migrated to the target database.

[0009] In one optional implementation, determining the migration statement based on the data to be migrated includes:

[0010] Obtain the log information corresponding to the data to be migrated in the source database;

[0011] Based on the log information, determine the operation statements related to the data to be migrated;

[0012] The migration statement is determined based on the operation statement.

[0013] In one optional implementation, determining the migration statement based on the operation statement includes:

[0014] Identify the sub-operation statements in the operation statements, wherein the sub-operation statements are used to instruct the creation of a data container in the source database to store the data to be migrated;

[0015] The sub-operation statement is identified as the migration statement.

[0016] In one optional implementation, migrating the target data to the target database includes:

[0017] Determine the syntax information of the target database;

[0018] The sub-operation statement is converted into a compatible operation statement that matches the syntax information;

[0019] Based on the compatibility operation statement, a data container is created in the target database to store the target data.

[0020] In one optional implementation, determining the compatibility information of the target database for the migration statement includes:

[0021] Obtain a set of compatibility relationships, and determine the compatibility relationships corresponding to the target database based on the set of compatibility relationships;

[0022] Based on the compatibility relationship, the migration statement that matches the target database is determined, and the compatibility information is determined based on the migration statement.

[0023] In one optional implementation, migrating the target data to the target database includes:

[0024] When the data to be migrated includes multiple sub-data, determine the sub-data set corresponding to the target data;

[0025] A corresponding migration thread is assigned to each sub-data in the subset dataset, so that each sub-data in the subset dataset can be migrated to the target database through the migration thread.

[0026] In one optional implementation, the method further includes:

[0027] Identify the target migration thread that failed to migrate in the migration thread, and determine the target sub-data corresponding to the target migration thread;

[0028] Determine the migration information of the target sub-data, and perform corresponding operations on the target sub-data based on the migration information.

[0029] In one optional implementation, performing the corresponding operation on the target sub-data based on the migration information includes:

[0030] When the migration information is determined to be the first migration information, the migration time of the target sub-data is determined, wherein the first migration information is used to indicate that the reason for the migration failure of the target sub-data is not related to the data type of the target sub-data;

[0031] The operation of migrating the target sub-data to the target database is performed according to the migration time.

[0032] In one optional implementation, performing corresponding data operations on the target sub-data based on the migration information includes:

[0033] When the migration information is determined to be second migration information, the data type of the target sub-data is determined, wherein the second migration information is used to indicate that the migration failure of the target sub-data is related to the data type of the target sub-data;

[0034] The compatibility relationship of the target database in the compatibility relationship set is updated based on the data type of the target sub-data, wherein the compatibility relationship is used to indicate the data types that the target database supports and does not support.

[0035] Secondly, embodiments of this disclosure also provide a database data processing apparatus, comprising:

[0036] An acquisition unit is used to acquire the data to be migrated and determine the migration statement based on the data to be migrated;

[0037] A determining unit is used to determine the compatibility information of the target database with the migration statement;

[0038] A migration unit is used to determine the target data corresponding to the compatibility information in the data to be migrated, and to migrate the target data to the target database.

[0039] Thirdly, embodiments of this disclosure also provide a computer device, including: a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, the steps of the first aspect above, or any possible implementation of the first aspect, are performed.

[0040] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the first aspect or any possible implementation of the first aspect.

[0041] In this embodiment, the data to be migrated is first obtained, and a migration statement is determined based on the data. This migration statement is used to migrate the data to a target database. Next, the compatibility information of the target database with the migration statement is determined, identifying the target data corresponding to this compatibility information within the data to be migrated. This target data is then migrated to the target database. This reduces migration obstacles and failures caused by data incompatibility during the migration process, ensuring the normal progress of data migration and improving overall data migration efficiency.

[0042] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0044] Figure 1 A flowchart of a database data processing method provided in an embodiment of this disclosure is shown;

[0045] Figure 2 This illustration shows a schematic diagram of the interaction between various functional modules in another database data processing method provided in an embodiment of the present disclosure;

[0046] Figure 3 This diagram illustrates a data processing system corresponding to a data processing method provided in an embodiment of the present disclosure;

[0047] Figure 4 A schematic diagram of a database data processing apparatus provided in an embodiment of the present disclosure is shown;

[0048] Figure 5 A schematic diagram of a computer device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0050] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0051] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0052] Research has shown that due to the diverse types of databases in distributed database systems, data migration between different databases is unavoidable. However, different types of databases often have differences in data types and syntax, leading to incompatibility. This incompatibility can cause migration obstacles and failures during the data migration process.

[0053] Based on the above research, this disclosure provides a database data processing method, apparatus, and computer device. In embodiments of this disclosure, data to be migrated is first acquired, and a migration statement is determined based on this data. This migration statement is used to migrate the data to a target database. Next, the compatibility information of the target database with the migration statement is determined, identifying the target data corresponding to this compatibility information within the data to be migrated. This target data is then migrated to the target database. This reduces migration obstacles and failures caused by data incompatibility during the migration process, ensuring the normal progress of data migration and improving overall data migration efficiency.

[0054] To facilitate understanding of this embodiment, a data processing method for a database disclosed in this disclosure will first be described in detail. The data processing method provided in this disclosure is generally executed by a computer device with certain computing capabilities. In some possible implementations, this data processing method can be implemented by a processor calling computer-readable instructions stored in memory.

[0055] See Figure 1 The diagram shows a flowchart of a database data processing method provided in an embodiment of this disclosure. The method includes steps S101 to S105, wherein:

[0056] S101: Obtain the data to be migrated, and determine the migration statement based on the data to be migrated.

[0057] In this embodiment of the disclosure, the data to be migrated can be data stored in the source database. When the data to be migrated is obtained, the data in the source database indicated by the received data migration instruction can be determined as the source data to be migrated, or the incremental data newly added in the source database can be determined as the data to be migrated, so as to synchronize the incremental data to the target database.

[0058] After identifying the data to be migrated, the corresponding migration statement can be determined. This migration statement can be a statement in the source database that operates on the data to be migrated, such as a create operation, query operation, modify operation, or delete operation. In particular, the create operation can be used to instruct the creation of a data container in the source database to store the data to be migrated, such as a data table. Based on this, the query operation, modify operation, and delete operation can be operations on the data table.

[0059] S103: Determine the compatibility information of the target database with the migration statement.

[0060] In this embodiment of the disclosure, a storage container, such as the data table, can be created in the target database based on the operation statements in the migration statement described above. However, since the source database and the target database are usually heterogeneous databases with differences in data types and syntax, for example, the source database may be an Oracle database and the target database may be a MySQL database. Therefore, the compatibility information of the target database with the migration statement can first be determined. This compatibility information can be used to indicate whether the target database is compatible with the migration statement. If the target database is not compatible with the migration statement, migration will be hindered or fail.

[0061] S105: Determine the target data corresponding to the compatibility information in the data to be migrated, and migrate the target data to the target database.

[0062] In this embodiment of the disclosure, there can be multiple migration statements. Compatibility information can be used to indicate migration statements that are compatible with the target database. Therefore, the target data corresponding to the migration statement in the data to be migrated can be determined. Specifically, the data table to be created by the migration statement can be determined, and the data in that data table can be identified as the target data.

[0063] Next, a migration task can be created based on the target data to migrate the target data to the target database. The specific migration process of this migration task is described below and will not be repeated here.

[0064] As described above, in this embodiment, the data to be migrated is first obtained, and a migration statement is determined based on this data. This migration statement is used to migrate the data to a target database. Next, the compatibility information of the target database with this migration statement is determined, identifying the target data corresponding to this compatibility information within the data to be migrated. This target data is then migrated to the target database. This reduces migration obstacles and failures caused by data incompatibility during the migration process, ensuring the normal progress of data migration and improving overall data migration efficiency.

[0065] In an optional implementation, step S101 above, which determines the migration statement based on the data to be migrated, specifically includes the following process:

[0066] S1011: Obtain the log information corresponding to the data to be migrated in the source database.

[0067] In this embodiment of the disclosure, log information of the data to be migrated in the source database can be collected by a collection module that is connected to the source database. The log information may include records of data read and write operations performed in the source database.

[0068] S1012: Based on the log information, determine the operation statements related to the data to be migrated.

[0069] In this embodiment of the disclosure, the log data may include operation statements related to the data to be migrated, wherein the operation statements may be operation statements such as creation operation, query operation, modification operation, deletion operation, etc.

[0070] S1013: Determine the migration statement based on the operation statement.

[0071] After identifying the above operation statements, migration statements can be identified from these operation statements. Specifically, at least some of the operation statements in these operation statements can be identified as migration statements to synchronize the operation records of the data to be migrated in the source database to the target database.

[0072] In this embodiment of the disclosure, log information in the source database can be obtained based on the acquisition module, and operation statements related to the data to be migrated can be determined based on the log information. Then, migration statements can be determined based on the operation statements, thereby providing a technical basis for data migration between the heterogeneous databases based on the migration statements.

[0073] In an optional implementation, step S1013 above, determining the migration statement based on the operation statement, specifically includes the following process:

[0074] (1) Determine the sub-operation statement in the operation statement, wherein the sub-operation statement is used to instruct the creation of a data container for the data to be migrated in the source database.

[0075] (2) The sub-operation statement is determined as the migration statement.

[0076] In this embodiment of the disclosure, the sub-operation statement can be the statement of the aforementioned creation operation. Specifically, the creation operation statement can be a DDL (Data Definition Language) operation statement. As can be seen from the above, the DDL operation statement can be used to instruct the creation of a data container, such as a data table, in the source database to store the data to be migrated. After the sub-operation statement is determined as a migration statement, a data container for storing the data to be migrated can be established in the target database based on the sub-operation statement.

[0077] Once the sub-operation statement is determined, the data to be migrated can be split based on the sub-operation statement to obtain multiple sub-data sets. For example, if the sub-operation statement corresponds to multiple storage operations, where each storage operation corresponds to at least one date, then the sub-operation statement can be used to instruct the data to be migrated to be stored in the data table according to the date. In this case, the data table can be split according to the date to obtain sub-data tables corresponding to each date, and the data stored in the sub-data tables can be identified as sub-data sets.

[0078] In this embodiment of the disclosure, the data to be migrated can be split into multiple sub-data based on sub-operation statements, and each sub-data can be migrated separately, thereby migrating the data to be migrated to the target database. In the case that the data to be migrated includes sub-data that cannot be migrated, it is possible to continue migrating other sub-data, thereby realizing the interruption resume in the data migration process.

[0079] In an optional implementation, step S103 above, determining the compatibility information of the target database with the migration statement, specifically includes the following process:

[0080] S1031: Obtain the compatibility relationship set, and determine the compatibility relationship corresponding to the target database based on the compatibility relationship set.

[0081] In this embodiment of the disclosure, a set of compatibility relationships can be pre-established, wherein the set of compatibility relationships may include the compatibility relationships between various heterogeneous databases and the source database in the above-mentioned distributed database system.

[0082] It should be understood that between heterogeneous databases, due to the differences in data types and syntax between different types of databases, the target database may be incompatible with the migration statements in the source database. For example, the source database and the target database may have significant syntax differences, or the data type of the data to be migrated indicated by the migration statement may not be included in the data types supported by the target database.

[0083] Based on this, the aforementioned compatibility relationship set can be established to record the compatibility relationships between various heterogeneous databases and the source database in the distributed database system. When the compatibility relationship between the heterogeneous database and the source database is incompatible, the reason for incompatibility can be recorded, such as syntax differences or data type differences.

[0084] S1032: Based on the compatibility relationship, determine the migration statement in the migration statement that matches the target database, and determine the compatibility information based on the migration statement.

[0085] In this embodiment of the disclosure, when there are multiple migration statements, each migration statement can correspond to a data table in the data to be migrated. Based on this, the compatibility relationship set can be searched based on the analysis and transformation module to determine the compatibility relationship between the target database and the source database, and the migration statement that can be compatible with the target database can be determined based on the compatibility relationship, so as to determine the compatibility information based on the migration statement.

[0086] In practice, when the compatibility relationship is compatible, compatibility information can be determined based on all compatible statements. When the compatibility relationship is incompatible, the reasons for incompatibility can be obtained. If the reason for incompatibility is a syntax difference, it can be determined that the target database is incompatible with the migration statement. If the reason for incompatibility is a data type difference, it can be determined that the data types supported by the target database are available, and a migration statement matching that data type can be identified within the migration statement. This migration statement is then identified as the one matching the target database. Next, compatibility information can be determined based on this migration statement, which includes the migration statement matching the target database—that is, the migration statement compatible with the target database.

[0087] In this embodiment of the disclosure, a migration statement compatible with the target database can be determined based on the compatibility relationship set, and compatibility information can be determined based on the migration statement, thereby reducing the problem of migration breakpoints caused by incompatibility of migration statements during the data migration process and further improving the overall migration efficiency of the data migration process.

[0088] In an optional implementation, step S105 above, which involves migrating the target data to the target database, specifically includes the following process:

[0089] S11: Determine the syntax information of the target database.

[0090] S12: Convert the sub-operation statement into a compatible operation statement that matches the syntax information.

[0091] S13: Based on the compatibility operation statement, create a data container in the target database to store the target data.

[0092] In this embodiment of the disclosure, a syntax transformation set can first be obtained, wherein the syntax transformation set includes syntax information between the heterogeneous database and the source database in the above-mentioned distributed database system, wherein the syntax information may include statement transformation rules, which are used to transform operation statements in the source database into operation statements that can be run in the target database.

[0093] As shown above, the sub-operation statement can be the statement corresponding to the creation operation mentioned above, that is, the statement used to create data containers such as data tables to store the data to be migrated in the source database. Therefore, based on the above syntax information, the sub-operation statement can be converted into a compatible operation statement so that the compatible operation statement conforms to the syntax of the target database, thereby enabling it to run in the target database.

[0094] In the process of converting a sub-operation statement into a compatible operation statement, several keywords within the sub-operation statement can first be identified. These keywords indicate the semantics of the sub-operation statement. Next, these keywords can be reorganized according to the syntax of the target database to obtain the compatible operation statement.

[0095] In this embodiment of the disclosure, sub-operation statements can be converted into compatible operation statements that can run in the target database based on the syntax information of the target database, thereby improving the efficiency and accuracy of statement conversion.

[0096] In an optional implementation, step S105 above, which involves migrating the target data to the target database, further includes the following process:

[0097] S21: When the data to be migrated includes multiple sub-data, determine the sub-data set corresponding to the target data.

[0098] S22: Assign a corresponding migration thread to each sub-data in the subset dataset, so as to migrate each sub-data in the subset dataset to the target database through the migration thread.

[0099] In this embodiment of the disclosure, the data to be migrated may include multiple data tables, and the data in the data tables is the aforementioned sub-data. Each data table has a corresponding migration statement. After determining the aforementioned compatibility information, the corresponding data table can be determined based on the migration statement in the compatibility information, and the sub-data set corresponding to the target data can be determined based on the data table.

[0100] Next, the distribution module can be used to assign a corresponding migration thread to each sub-data in the subset. Specifically, the migration thread can create a data table corresponding to the sub-data in the target database based on the migration statement corresponding to the sub-data, and store the sub-data based on the data table.

[0101] It should be understood that when migrating data through migration threads, multiple migration threads can be processed in parallel, and each migration thread can actively report the data migration results to determine whether the corresponding sub-data has been successfully migrated.

[0102] In this embodiment of the disclosure, multiple migration threads can be used to migrate the corresponding sub-data separately to achieve parallel processing during the data migration process, thereby improving processing efficiency. In addition, each migration thread can actively report the data migration result to determine whether the corresponding sub-data has been successfully migrated based on the data migration result, thereby facilitating the monitoring of the data migration process.

[0103] In an optional implementation, step S105 above further includes the following process:

[0104] S31: Determine the target migration thread that failed to migrate in the migration thread, and determine the target sub-data corresponding to the target migration thread.

[0105] S32: Determine the migration information of the target sub-data, and perform corresponding operations on the target sub-data based on the migration information.

[0106] In this embodiment of the disclosure, due to potential database updates or other issues in the heterogeneous databases within the aforementioned distributed database system, the migration statements corresponding to some migration threads may become incompatible with the target database, leading to migration failure for those threads. Based on this, the target migration thread that failed can be identified, along with the target sub-data corresponding to that thread. Specifically, a re-acquisition command can be initiated to allow the aforementioned acquisition module to retrieve the target sub-data again from the source database, facilitating subsequent operations on that target sub-data.

[0107] Next, the migration information of the first sub-migrated data can be determined. This migration information is used to indicate the reason for the failure of the target sub-data migration. Specifically, the migration information may include the migration log of the first sub-migrated data during the migration process.

[0108] It should be understood that the actions taken on the first sub-migrated data can vary depending on the reason for the migration failure. For example, the action could be to re-initiate the migration operation on the first sub-migrated data, or not to initiate the migration operation on the first sub-migrated data at all. The specific process for performing the operation on the first sub-migrated data is described below.

[0109] In this embodiment of the disclosure, considering that there may be target migration threads that fail to migrate in the migration thread, the migration information of the target sub-data corresponding to the target migration thread can be determined, so as to determine the reason for the failure of the target sub-data migration based on the migration information, thereby adopting different operation strategies for the first sub-migration data, and thus improving the processing scheme for the failed migration data in this disclosure.

[0110] In an optional implementation, step S32 above, which involves performing corresponding operations on the target sub-data based on the migration information, specifically includes the following process:

[0111] (1) When the migration information is determined to be the first migration information, the migration time of the target sub-data is determined, wherein the first migration information is used to indicate that the reason for the migration failure of the target sub-data is not related to the data type of the target sub-data;

[0112] (2) Perform the operation of migrating the target sub-data to the target database according to the migration time.

[0113] In this embodiment of the disclosure, the first migration information mentioned above can be used to indicate that the reason for the failure of the migration of the target sub-data is not a data type incompatibility problem, that is, the reason for the failure of the migration of the target sub-data is not related to the data type of the target sub-data. Specifically, the reason for the failure can be a migration interruption caused by network fluctuations or other reasons.

[0114] After identifying the migration information as the primary migration information, the task of migrating this target sub-data can be suspended, and the migration time for this target sub-data can be determined. For example, this migration time can be when the migration of the remaining sub-data in the target data is completed. After this migration time is reached, a migration thread can be allocated for this target sub-data in the migration thread pool, so that the target sub-data can be migrated to the target database through this migration thread.

[0115] In this embodiment of the disclosure, when the failure of the target sub-data migration is not due to data type incompatibility, a migration time can be determined for the target sub-data, and the migration operation can be performed again at the migration time to migrate the target sub-data to the target database, thereby improving the integrity of the migrated target data.

[0116] In an optional implementation, step S32 above, which involves performing corresponding operations on the target sub-data based on the migration information, further includes the following process:

[0117] (1) When the migration information is determined to be the second migration information, the data type of the target sub-data is determined, wherein the second migration information is used to indicate that the migration failure of the target sub-data is related to the data type of the target sub-data;

[0118] (2) Based on the data type of the target sub-data, update the compatibility relationship corresponding to the target database in the compatibility relationship set.

[0119] In this embodiment of the disclosure, the second migration information can be used to indicate that the reason for the failure of the target sub-data migration is data type incompatibility, that is, the target database is incompatible with the data type of the target sub-data.

[0120] When the above migration information is determined to be the second migration information, the above set of compatibility relationships can be obtained, and the compatibility relationships corresponding to the target data can be updated. Specifically, the data type of the target sub-data can be updated to the incompatibility reason corresponding to the compatibility relationship.

[0121] Simultaneously, this second migration information can be sent to the early warning module, enabling managers to control the overall migration process and understand the reasons for the failure of the target sub-data migration, thereby iterating the data migration strategy and improving the success rate of data migration.

[0122] In this embodiment of the disclosure, when the failure of the target sub-data migration is due to data type incompatibility, that is, when the target database is incompatible with the data type of the target sub-data, the compatibility relationship set can be updated based on the data type, thereby reducing the number of migration failures during the data migration process and improving the overall migration efficiency of the data migration process.

[0123] See Figure 2 The diagram illustrates the interaction between functional modules in another database data processing method provided in this embodiment of the present disclosure. These functional modules may include: a data acquisition module, a distribution module, an analysis and transformation module, a rule module, an alarm module, and a playback module. The method includes steps S201 to S206, wherein:

[0124] S201: The above-mentioned acquisition module obtains the data to be migrated from the source database, as well as the log information corresponding to the data to be migrated. The log information includes operation statements related to the data to be migrated.

[0125] In this embodiment of the disclosure, a migration statement can be determined based on an operation statement. Specifically, the DDL operation statement in the operation statement can be obtained and used as the migration statement.

[0126] S202: Through the above distribution module, a corresponding migration thread is assigned to each sub-data in the above-mentioned data to be migrated, so as to perform the migration operation of the corresponding sub-data.

[0127] In this embodiment of the disclosure, the above-mentioned compatible dataset can be obtained first, and the compatibility relationship corresponding to the target database can be obtained based on the compatible dataset. Based on the compatibility relationship, the migration statements of the sub-data corresponding to each migration thread can be converted into the above-mentioned compatible operation statements, and the target migration threads that cannot be converted into compatible operation statements can be determined.

[0128] S203: The analysis and transformation module obtains the above-mentioned compatibility relationship set from the above-mentioned rule module, and determines the compatibility relationship corresponding to the above-mentioned target database based on the compatibility relationship set.

[0129] S204: Based on the above compatibility relationship, identify the target data that is compatible with the target database and / or the non-target data that is not compatible with the target database in the data to be migrated.

[0130] In this embodiment of the disclosure, during the process of determining non-target data, the distribution module can obtain the non-target data that the acquisition module has re-retrieved from the source database based on the re-acquisition instruction.

[0131] S205: Through the above playback module, data playback operation is performed in the target database based on compatible operation statements to store the target data in the target database.

[0132] S206: If non-target data that is incompatible with the target database is identified in the data to be migrated, the data type of the non-target data is determined, and the compatibility relationship of the target database in the above compatibility relationship set is updated based on the data type through the alarm module mentioned above.

[0133] In this embodiment of the disclosure, the process of updating the compatibility relationship corresponding to the target database in the above-mentioned compatibility relationship set is as described in the embodiment corresponding to step S32 above, and will not be repeated here.

[0134] In summary, in this embodiment, the data to be migrated is first obtained, and a migration statement is determined based on this data. This migration statement is used to migrate the data to a target database. Next, the compatibility information of the target database with this migration statement is determined, identifying the target data corresponding to this compatibility information within the data to be migrated. This target data is then migrated to the target database. This reduces migration obstacles and failures caused by data incompatibility during the migration process, ensuring the normal progress of data migration and improving overall data migration efficiency.

[0135] Based on the same inventive concept, this disclosure also provides a data processing system corresponding to the data processing method, see below. Figure 3 As shown, the data processing system includes: a data acquisition module, a distribution module, an analysis and transformation module, a rules module, a playback module, and an alarm module; wherein,

[0136] The data acquisition module is used to obtain the data to be migrated from the source database, as well as the corresponding log information.

[0137] The distribution module is used to allocate corresponding migration threads to each sub-data in the above-mentioned data to be migrated, so as to perform the migration operation of the corresponding sub-data.

[0138] The analysis and transformation module is used to obtain the compatibility relationship set from the rule module, determine the compatibility relationship corresponding to the target database based on the compatibility relationship set, and determine the target data that is compatible with the target database and / or the non-target data that is not compatible with the target database in the data to be migrated based on the compatibility relationship.

[0139] The rules module is used to store the above set of compatibility relationships.

[0140] The playback module is used to perform data playback operations in the target database based on compatible operation statements, so as to store the target data in the target database.

[0141] The alarm module is used to determine the data type of the non-target data and, based on the data type, update the compatibility relationship corresponding to the target database in the above compatibility relationship set.

[0142] In this embodiment, the data to be migrated is first obtained, and a migration statement is determined based on the data. This migration statement is used to migrate the data to a target database. Next, the compatibility information of the target database with the migration statement is determined, identifying the target data corresponding to this compatibility information within the data to be migrated. This target data is then migrated to the target database. This reduces migration obstacles and failures caused by data incompatibility during the migration process, ensuring the normal progress of data migration and improving overall data migration efficiency.

[0143] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0144] Based on the same inventive concept, this disclosure also provides a data processing device corresponding to the data processing method. Since the principle of the device in this disclosure for solving the problem is similar to that of the data processing method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0145] Reference Figure 4 The diagram shown is a schematic representation of a database data processing apparatus according to an embodiment of this disclosure. The apparatus includes: an acquisition unit 41, a determination unit 42, and a migration unit 43; wherein,

[0146] Acquisition unit 41 is used to acquire the data to be migrated and determine the migration statement based on the data to be migrated;

[0147] The determining unit 42 is used to determine the compatibility information of the target database with the migration statement;

[0148] Migration unit 43 is used to determine the target data corresponding to the compatibility information in the data to be migrated, and migrate the target data to the target database.

[0149] In this embodiment, the data to be migrated is first obtained, and a migration statement is determined based on the data. This migration statement is used to migrate the data to a target database. Next, the compatibility information of the target database with the migration statement is determined, identifying the target data corresponding to this compatibility information within the data to be migrated. This target data is then migrated to the target database. This reduces migration obstacles and failures caused by data incompatibility during the migration process, ensuring the normal progress of data migration and improving overall data migration efficiency.

[0150] In one possible implementation, the acquisition unit 41 is further configured to:

[0151] Obtain the log information corresponding to the data to be migrated in the source database;

[0152] Based on the log information, determine the operation statements related to the data to be migrated;

[0153] The migration statement is determined based on the operation statement.

[0154] In one possible implementation, the acquisition unit 41 is further configured to:

[0155] Identify the sub-operation statements in the operation statements, wherein the sub-operation statements are used to instruct the creation of a data container in the source database to store the data to be migrated;

[0156] The sub-operation statement is identified as the migration statement.

[0157] In one possible implementation, the migration unit 43 is further configured to:

[0158] Determine the syntax information of the target database;

[0159] The sub-operation statement is converted into a compatible operation statement that matches the syntax information;

[0160] Based on the compatibility operation statement, a data container is created in the target database to store the target data.

[0161] In one possible implementation, the determining unit 42 is further configured to:

[0162] Obtain a set of compatibility relationships, and determine the compatibility relationships corresponding to the target database based on the set of compatibility relationships;

[0163] Based on the compatibility relationship, the migration statement that matches the target database is determined, and the compatibility information is determined based on the migration statement.

[0164] In one possible implementation, the migration unit 43 is further configured to:

[0165] When the data to be migrated includes multiple sub-data, determine the sub-data set corresponding to the target data;

[0166] A corresponding migration thread is assigned to each sub-data in the subset dataset, so that each sub-data in the subset dataset can be migrated to the target database through the migration thread.

[0167] In one possible implementation, the migration unit 43 is further configured to:

[0168] Identify the target migration thread that failed to migrate in the migration thread, and determine the target sub-data corresponding to the target migration thread;

[0169] Determine the migration information of the target sub-data, and perform corresponding operations on the target sub-data based on the migration information.

[0170] In one possible implementation, the migration unit 43 is further configured to:

[0171] When the migration information is determined to be the first migration information, the migration time of the target sub-data is determined, wherein the first migration information is used to indicate that the reason for the migration failure of the target sub-data is not related to the data type of the target sub-data;

[0172] The operation of migrating the target sub-data to the target database is performed according to the migration time.

[0173] In one possible implementation, the migration unit 43 is further configured to:

[0174] When the migration information is determined to be second migration information, the data type of the target sub-data is determined, wherein the second migration information is used to indicate that the migration failure of the target sub-data is related to the data type of the target sub-data;

[0175] The compatibility relationship of the target database in the compatibility relationship set is updated based on the data type of the target sub-data, wherein the compatibility relationship is used to indicate the data types that the target database supports and does not support.

[0176] The description of the processing flow of each unit in the device and the interaction flow between each unit can be found in the relevant descriptions in the above method embodiments, and will not be detailed here.

[0177] Corresponding to Figure 1 In addition to the data processing methods described herein, this disclosure also provides a computer device 500, such as... Figure 5 The diagram shown is a structural schematic of a computer device 500 provided in an embodiment of this disclosure, including:

[0178] The system includes a processor 51, a memory 52, and a bus 53. The memory 52 stores execution instructions and includes main memory 521 and external memory 522. The main memory 521, also called internal memory, temporarily stores the processing data in the processor 51, as well as data exchanged with external memory such as a hard disk. The processor 51 exchanges data with the external memory 522 through the main memory 521. When the computer device 500 is running, the processor 51 communicates with the memory 52 through the bus 53, causing the processor 51 to execute the following instructions:

[0179] Obtain the data to be migrated, and determine the migration statement based on the data to be migrated;

[0180] Determine the compatibility information of the target database with the migration statement;

[0181] The target data corresponding to the compatibility information in the data to be migrated is determined, and the target data is migrated to the target database.

[0182] This disclosure also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of the data processing method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0183] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the data processing method described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0184] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0185] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0186] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0188] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A data processing method for a database, characterized in that, include: Obtain the data to be migrated, and determine the migration statement based on the data to be migrated, wherein the migration statement is a statement in the source database that operates on the data to be migrated; Determining the compatibility information of the target database for the migration statement includes: obtaining a set of compatibility relationships, and determining the compatibility relationship corresponding to the target database based on the set of compatibility relationships; determining the migration statement that matches the target database in the migration statement based on the compatibility relationship, and determining the compatibility information based on the migration statement; The step of determining the migration statements that match the target database based on the compatibility relationship includes: When the compatibility relationship is incompatible, obtain the reason for the incompatibility; When the incompatibility is due to syntax differences, it is determined that the target database is incompatible with the migration statement; When the incompatibility is due to data type differences, the data types supported by the target database are determined, and the migration statement whose data type matches the data type supported by the target database is determined as the migration statement that matches the target database. The target data corresponding to the compatibility information in the data to be migrated is determined, and the target data is migrated to the target database, wherein the compatibility information includes migration statements that are compatible with the target database.

2. The method according to claim 1, characterized in that, The step of determining the migration statement based on the data to be migrated includes: Obtain the log information corresponding to the data to be migrated in the source database; Based on the log information, determine the operation statements related to the data to be migrated; The migration statement is determined based on the operation statement.

3. The method according to claim 2, characterized in that, Determining the migration statement based on the operation statement includes: Identify the sub-operation statements in the operation statements, wherein the sub-operation statements are used to instruct the creation of a data container in the source database to store the data to be migrated; The sub-operation statement is identified as the migration statement.

4. The method according to claim 3, characterized in that, The migration of the target data to the target database includes: Determine the syntax information of the target database; The sub-operation statement is converted into a compatible operation statement that matches the syntax information; Based on the compatibility operation statement, a data container is created in the target database to store the target data.

5. The method according to claim 1, characterized in that, The migration of the target data to the target database includes: When the data to be migrated includes multiple sub-data, determine the sub-data set corresponding to the target data; A corresponding migration thread is assigned to each sub-data in the subset dataset, so that each sub-data in the subset dataset can be migrated to the target database through the migration thread.

6. The method according to claim 5, characterized in that, The method further includes: Identify the target migration thread that failed to migrate in the migration thread, and determine the target sub-data corresponding to the target migration thread; Determine the migration information of the target sub-data, and perform corresponding operations on the target sub-data based on the migration information.

7. The method according to claim 6, characterized in that, The operation performed on the target sub-data based on the migration information includes: When the migration information is determined to be the first migration information, the migration time of the target sub-data is determined, wherein the first migration information is used to indicate that the reason for the migration failure of the target sub-data is not related to the data type of the target sub-data; The operation of migrating the target sub-data to the target database is performed according to the migration time.

8. The method according to claim 6, characterized in that, The step of performing corresponding data operations on the target sub-data based on the migration information includes: When the migration information is determined to be second migration information, the data type of the target sub-data is determined, wherein the second migration information is used to indicate that the migration failure of the target sub-data is related to the data type of the target sub-data; The compatibility relationship of the target database in the compatibility relationship set is updated based on the data type of the target sub-data, wherein the compatibility relationship is used to indicate the data types that the target database supports and does not support.

9. A data processing device for a database, characterized in that, include: An acquisition unit is used to acquire data to be migrated and determine migration statements based on the data to be migrated, wherein the migration statements are statements in the source database that operate on the data to be migrated; A determining unit is configured to determine the compatibility information of the target database for the migration statement, including: obtaining a set of compatibility relationships and determining the compatibility relationship corresponding to the target database based on the set of compatibility relationships; determining the migration statement that matches the target database in the migration statement based on the compatibility relationship, and determining the compatibility information based on the migration statement; The step of determining the migration statements that match the target database based on the compatibility relationship includes: When the compatibility relationship is incompatible, obtain the reason for the incompatibility; When the incompatibility is due to syntax differences, it is determined that the target database is incompatible with the migration statement; When the incompatibility is due to data type differences, the data types supported by the target database are determined, and the migration statement whose data type matches the data type supported by the target database is determined as the migration statement that matches the target database. A migration unit is used to determine the target data corresponding to the compatibility information in the data to be migrated, and to migrate the target data to the target database, wherein the compatibility information includes migration statements compatible with the target database.

10. A computer device, characterized in that, include: The computer device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the data processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the data processing method as described in any one of claims 1 to 8.

Images