Data processing methods, apparatus, electronic devices and storage media

Abstract

The data processing method, apparatus, electronic device, and storage medium of this application acquire a first entity class of a first data table to be migrated in a database to be migrated, and a second entity class of a second data table in a migration database; generate at least one first annotation based on the operation information of the first data table; create a migration task based on the first entity class, the second entity class, and the first annotation, the migration task being used to migrate data from the first data table to the second data table; through the above method, when the structure of the data table in the migration database is different from the structure in the data to be migrated, it can be characterized based on the first annotation generated based on the content of the field to be changed, and then the corresponding function can be called according to the operation type code in the first annotation to realize the conversion operation of the corresponding field, which is beneficial to improving the code reusability of the migration task, and when an error occurs in the data structure conversion, it can be traced through the corresponding first annotation, thereby improving the efficiency of data migration.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Data migration between different databases is very common in the current software industry. For example, when upgrading a system, it is often necessary to change the backend database due to cost or performance considerations, such as changing Oracle database to MySQL database, MySQL database to PostgreSQL database, PostgreSQL database to Oracle database, and so on.

[0003] In existing technologies, when migrating data tables from an old database to a new database, it is sometimes necessary to transform the data table structure during the data migration process to optimize the data table structure, thereby speeding up response and improving development efficiency.

[0004] To achieve data structure transformation, existing technologies typically rely on procedural code written by programmers, which has poor reusability. Furthermore, it is difficult to trace the source when errors occur during data migration and data structure transformation, which is not conducive to improving the efficiency of data migration. Summary of the Invention

[0005] The purpose of this application is to provide a data processing method, apparatus, electronic device, and storage medium to solve the technical problem in the prior art that the data migration efficiency is not improved when converting the data table structure during the data migration process.

[0006] The technical solution of this application is as follows: A data processing method is provided, including:

[0007] Retrieve the first entity class of the first data table to be migrated in the database being migrated, and the second entity class of the second data table in the migration database;

[0008] At least one first annotation is generated based on the operation information of the first data table, wherein the operation information is used to indicate the content of the field to be changed in the second data table relative to the first data table, each first annotation corresponds to a second field in the second data table, and each first annotation includes an operation type code and a first field in the first data table, wherein the data of the second field is obtained based on the data of the first field;

[0009] A migration task is created based on the first entity class, the second entity class, and the at least one first annotation. The migration task is used to migrate data from the first data table in the migrated database to the second data table in the migration database.

[0010] In some optional implementations, after creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the method further includes:

[0011] At least one second annotation is generated based on the verification information of the second data table, wherein each second annotation includes a verification type code;

[0012] Add the at least one second annotation to the migration task.

[0013] In some optional implementations, after creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the method further includes:

[0014] At least one third annotation is generated based on the foreign key association information of the second data table, wherein each third annotation includes foreign key migration code, a third field in the first data table that serves as the primary key, and a fourth field that serves as the foreign key and is associated with the third field.

[0015] Retrieve the first annotation corresponding to the fourth field;

[0016] If the acquisition is successful, the third annotation is added to the first annotation of the migration task corresponding to the fourth field;

[0017] If the retrieval fails, a corresponding first annotation is generated based on the second field corresponding to the fourth field in the second data table and the third annotation.

[0018] In some optional implementations, after creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the method further includes:

[0019] The migration task is executed to migrate data from the first data table to the second data table.

[0020] In some optional implementations, performing the migration task to migrate data from the first data table to the second data table includes:

[0021] Determine whether the current field in the first data table is the field to be changed;

[0022] When it is determined that the current field is a field to be changed, according to the first annotation corresponding to the current field, the data of the current field is processed according to the operation type code, and the processed data is migrated to the second field of the second data table;

[0023] When it is determined that the current field is not a field to be changed, the data of the current field is migrated to the second data table.

[0024] In some optional implementations, after determining that the current field is a field to be changed, performing the processing corresponding to the operation type code on the data of the current field according to the first annotation corresponding to the current field, and migrating the processed data to the second field of the second data table, the method further includes:

[0025] According to the third annotation, the fourth field is associated with the second field as a primary key, thereby removing the primary and foreign key association between the fourth field and the corresponding field of the third field in the second data table.

[0026] In some optional implementations, after determining that the current field is a field to be changed, performing the processing corresponding to the operation type code on the data of the current field according to the first annotation corresponding to the current field, and migrating the processed data to the second field of the second data table, the method further includes:

[0027] Based on the second annotation, perform the processing corresponding to the validation type code on the data of all fields in the second data table.

[0028] Another technical solution of this application is as follows: A data processing apparatus is provided, comprising:

[0029] Create a module to obtain the first entity class of the first data table to be migrated in the database being migrated, and the second entity class of the second data table in the migration database;

[0030] An annotation module is used to generate at least one first annotation based on the operation information of the first data table, wherein the operation information is used to indicate the content of the field to be changed in the second data table relative to the first data table, each first annotation corresponds to a second field in the second data table, and each first annotation includes an operation type code and a first field in the first data table, wherein the data of the second field is obtained based on the data of the first field;

[0031] The task module is used to create a migration task based on the first entity class, the second entity class, and the at least one first annotation. The migration task is used to migrate data from the first data table in the migrated database to the second data table in the migration database.

[0032] Another technical solution of this application is as follows: An electronic device is provided, including a processor and a memory coupled to the processor, the memory storing program instructions that can be executed by the processor; when the processor executes the program instructions stored in the memory, it implements the above-mentioned data processing method.

[0033] Another technical solution of this application is as follows: a storage medium is provided, wherein program instructions are stored in the storage medium, and when the program instructions are executed by a processor, they can implement the above-mentioned data processing method.

[0034] This application discloses a data processing method, apparatus, electronic device, and storage medium. The method involves obtaining a first entity class of a first data table to be migrated in a database to be migrated, and a second entity class of a second data table in a migration database. It then generates at least one first annotation based on operation information from the first data table. This operation information indicates the content of a field to be modified in the second data table relative to the first data table. Each first annotation corresponds to a second field in the second data table, and each first annotation includes an operation type code and the first field in the first data table. The data of the second field is obtained based on the data of the first field. Finally, a migration task is created based on the first entity class, the second entity class, and the at least one first annotation. This migration task is used to migrate data from the first data table in the database to the second data table in the migration database. By means of the above method, when the structure of a data table in the migration database differs from its structure in the data to be migrated, it can be characterized by the first annotation generated based on the content of the field to be modified. Furthermore, the operation type code in the first annotation can be used to call the corresponding function to implement the conversion operation of the corresponding field, which improves the code reusability of the migration task. Additionally, when an error occurs during data structure conversion, it can be traced back to the source through the corresponding first annotation, thereby improving the efficiency of data migration. Attached Figure Description

[0035] Figure 1 This is a flowchart of a data processing method according to an embodiment of this application;

[0036] Figure 2 This is a flowchart of a sub-step in a data processing method according to an embodiment of this application;

[0037] Figure 3 This is an example diagram of a migration task in a data processing method according to an embodiment of this application;

[0038] Figure 4 This is a schematic diagram of the structure of a data processing apparatus according to an embodiment of this application;

[0039] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the structure of a storage medium according to an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0042] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0043] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0044] One embodiment of this application provides a data processing method. The execution entity of the data processing method includes, but is not limited to, at least one of electronic devices, such as a server or a terminal, that can be configured to execute the data processing method provided in this application embodiment. In other words, the data processing method can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0045] Please see Figure 1 The diagram shown is a schematic flowchart of a data processing method provided in one embodiment of this application. It should be noted that if substantially the same result is achieved, the method of this application does not necessarily require further elaboration. Figure 1 The illustrated process sequence is limited. In this embodiment, the data processing method includes the following steps:

[0046] S101, obtain the first entity class of the first data table to be migrated in the database to be migrated and the second entity class of the second data table in the migration database.

[0047] In this system, the database being migrated is the old database, and the database being migrated is the new database. Entity classes are used to represent the corresponding data tables; specifically, a first entity class represents a first data table, which is stored in the database being migrated, and a second entity class represents a second data table, which is stored in the database being migrated. In an optional implementation, the entity class can be the name of the corresponding data table. For example, if a first data table named RmsUserPersonalized is migrated from the database being migrated to a second data table named SysUserPermission in the database being migrated, then the first entity class would be RmsUserPersonalized, and the second entity class would be SysUserPermission.

[0048] S102, generate at least one first annotation based on the operation information of the first data table, wherein the operation information is used to indicate the content of the field to be changed in the second data table relative to the first data table, each first annotation corresponds to a second field in the second data table, each first annotation includes an operation type code and a first field in the first data table, and the data of the second field is obtained based on the data of the first field.

[0049] The fields to be changed include the fields in the first data table that need to be converted and the conversion method. The operation type code in the first annotation represents the data conversion operation and is the identifier of the corresponding operation. For example, if the userId field in the first data table named RmsUserPersonalized needs to be renamed to id during data migration, the operation type code in the generated first annotation would be @ChangeName. This operation type code is associated with the program code that modifies the field name. The first field is userId, and the second field is id. When the field name is modified, the data in the second field id is directly copied from the data in the first field userId. As another example, if the amount field in the first data table named RmsUserPersonalized needs to have range labels added to its data based on the size of the corresponding data during data migration, with values ​​greater than a first threshold corresponding to the first range, values ​​less than or equal to the first threshold but greater than the second threshold corresponding to the second range, and values ​​less than or equal to the second threshold corresponding to the third range, the operation type code in the first annotation would be @addFlag. This operation type code is associated with the program code that adds the corresponding label based on the data. The first field is amount, and the second field is also amount. The data in the second field is obtained by adding the corresponding range label to the data in the first field. In this embodiment, fields that do not require data transformation are not configured with corresponding first annotations.

[0050] In one alternative implementation, the first field and the second field can be different or the same. For example, when modifying the field name, the first field and the second field are different; or when adding data labels, the first field and the second field are the same.

[0051] In one optional implementation, the functional program code corresponding to each operation type code is used to implement a function, such as modifying a field name or adding a label. This functional program code can be pre-written and stored in a code library. Each functional program code corresponds to an operation type code, and the corresponding functional program code can be directly obtained from the code library through the operation type code. The functional program codes in this application have strong reusability. For different fields in different data tables, the same functional program code can be associated with the first annotation.

[0052] S103, a migration task is created based on the first entity class, the second entity class, and the at least one first annotation. The migration task is used to migrate the data of the first data table in the migrated database to the second data table in the migration database.

[0053] Specifically, a migration task can be created for each second data table that needs data transformation. Annotations are added to the second entity class corresponding to the second data table. Each first annotation under the second entity class corresponds to a second field that needs data transformation. The second field is a field in the second data table. For each first annotation, the source data before the operation is obtained based on the first field, the corresponding functional program code is obtained based on the operation type code, and the position of the data after the operation in the second data table is determined based on the second field.

[0054] By adopting the technical solution of this application embodiment, when the structure of the data table in the migration database is different from the structure in the data being migrated, it can be characterized by the first annotation generated based on the content of the field to be changed. Then, the corresponding function can be called according to the operation type code in the first annotation to realize the conversion operation of the corresponding field. This is beneficial to improve the code reusability of the migration task. Furthermore, when an error occurs in the data structure conversion, it can be traced through the corresponding first annotation, thereby improving the efficiency of data migration.

[0055] Furthermore, the technical solution adopted in this application embodiment can be applied to the migration of data between two databases, and is not limited by the difference between the structure of the data table before and after the migration. The data migration method is more flexible and easier to use.

[0056] In one optional implementation, please refer to Figure 2 As shown, the following steps are included after step S103:

[0057] S201, generate at least one second annotation based on the verification information of the second data table, wherein each second annotation includes a verification type code;

[0058] S202, add the at least one second annotation to the migration task.

[0059] In addition, a verification operation can be added to the migration task to verify the data in the second data table obtained after data migration. For example, the verification information may include verifying the data completeness of all fields in the second data table, removing duplicate associations in all data in the second data table, or verifying that the data in all data in the second data table does not need to be migrated. When the second data table contains data corresponding to fields that do not need to be migrated, the corresponding data will be deleted.

[0060] This involves writing corresponding validation function code for each validation type, and associating the validation function code with a validation type code to characterize it. For example, the validation type code for "no migration required" is `@FieldIgnore`; and the validation type code for "removing empty dirty data" is `@RemoveNull`. In this implementation, at least one second annotation can be added after all first annotations. In each second annotation, the corresponding validation function code is retrieved from the code library based on the validation type code. Alternatively, the second annotation can be added to the first annotation containing the second field that needs to be validated.

[0061] In an optional implementation, step S103 is followed by the following steps:

[0062] S301, generate at least one third annotation based on the foreign key association information of the second data table, wherein each third annotation includes foreign key migration code, a third field in the first data table as a primary key, and a fourth field associated with the third field as a foreign key;

[0063] S302, Obtain the first annotation corresponding to the fourth field;

[0064] S303, if successful, the third annotation is added to the first annotation of the migration task corresponding to the fourth field;

[0065] S304, if the acquisition fails, then generate the corresponding first annotation based on the second field corresponding to the fourth field in the second data table and the third annotation.

[0066] The foreign key migration code is used to migrate foreign key relationships in the second data table. This code represents the program code that implements the foreign key migration operation, linking the foreign key originally associated with a third field in the first data table to a second field in the second data table after the data migration. The foreign key migration code is `@ForeignKeyMigration`.

[0067] In step S302, the first annotation corresponding to the fourth field is the first annotation of the second field to which a primary-foreign key relationship is to be established with the fourth field. Step S302 specifically includes: determining the second field in the second data table to which a primary-foreign key relationship is to be established with the fourth field; and obtaining the first annotation containing the second field. If the acquisition is successful, it means that the second field needs to undergo other data transformations besides foreign key migration, and the third annotation can be directly added to the first annotation of the second field that needs to be associated with the fourth field as a primary key. If the acquisition fails, it means that the second field does not need to undergo other data transformations besides foreign key migration, and the first annotation corresponding to the foreign key migration operation is generated based on the third annotation and the corresponding second field.

[0068] Taking the migration of the RMS_USER_PERSONALIZED table from an Oracle database to the sys_user_permission table from a MySQL database as an example, see the migration task details below. Figure 3 As shown, the first annotation was generated for the second field - user id and the second field - permission id, which require data transformation.

[0069] In an optional implementation, step S103 is followed by the following steps:

[0070] S401, execute the migration task to migrate the data from the first data table to the second data table.

[0071] In some implementations, step S401 specifically includes the following steps:

[0072] S501, determine whether the current field of the first data table is a field to be changed;

[0073] Specifically, the first annotation can be used to determine whether the current field is a field to be changed. If the current field has a corresponding first annotation, it is a field to be changed; if the current field does not have a corresponding first annotation, it is not a field to be changed.

[0074] S502, when it is determined that the current field is a field to be changed, according to the first annotation corresponding to the current field, the data of the current field is processed according to the operation type code, and the processed data is migrated to the second field of the second data table;

[0075] For fields to be changed, data migration is performed based on the first annotation.

[0076] S503, when it is determined that the current field is not a field to be changed, the data of the current field is migrated to the second data table;

[0077] For fields that do not require modification, data migration can be performed directly.

[0078] In some implementations, after step S502, the following steps are also included:

[0079] S601, according to the third annotation, the fourth field is associated with the second field as a primary key, and the primary-foreign key association between the fourth field and the corresponding field of the third field in the second data table is released.

[0080] In some implementations, after step S502, the following steps are also included:

[0081] S701, based on the second annotation, perform processing on all fields of the second data table corresponding to the validation type code.

[0082] This application can be applied to scenarios where Oracle databases are being replaced with MySQL databases. Firstly, for development, relying solely on procedural code written by individual programmers leads to the following problems: First, low development efficiency: procedural code has poor readability, is difficult to reuse, and is difficult to modify. Second, prone to errors: lacking unified validation and error handling, it is prone to errors, and when problems occur, it is difficult to trace the source. Third, difficult to extend: adding new features is limited by the table structure, and without conversion tools, many optimization features are difficult to implement. Secondly, for business users: the migrated system often needs to be compatible with the old system's data, but the lack of easy-to-use tools for migrating old data leads to low development efficiency, excessively long migration cycles, and negatively impacts the business experience. Especially when oversights or errors occur during the migration of old data, it can cause production losses. Businesses often hope that the new system will improve efficiency and have higher availability. However, if the structure of the old data cannot be optimized, it often becomes a bottleneck, affecting the user experience.

[0083] The data processing method described in this application simplifies the development process. After integration, it mainly includes three steps: First, annotate the corresponding fields of the entity class; second, create a new migration task based on the annotations; and finally, pass the entity class into the migration method. This achieves data migration and transformation. The following are the achievable functions and future directions for functional expansion: First, by configuring annotations, common functions can be reused, reducing code volume and improving code structure. For example, to change a field name, use the `@ChangeName` annotation. For example, if a new primary key is generated, corresponding to a foreign key in the associated table, the `@MigrationForeignKey` annotation can be used for migration. Second, it strongly validates the completeness of migration fields. If a field does not need to be migrated, it should be ignored using `@FieldIgnore`. Third, it supports filtering dirty data, such as empty fields or duplicate relationships. This can be flexibly configured as needed. Fourth, it supports the migration of large amounts of data, automatically performing pagination, processing 1000 rows at a time by default, and this is configurable. Fifth, it supports custom data migration methods, avoiding limitations imposed by the framework itself and enabling more flexible functionality.

[0084] For developers, the data processing method described in this application offers the following benefits: First, it is simple and easy to use: annotated fields are used, and new tasks are created by passing in the entity class. Second, it improves development efficiency: not only is development convenient, but problems and new requirements are also easier to locate and modify. Third, it offers flexible configuration: to achieve the desired transformation, the corresponding annotation is used on the corresponding entity field. Custom transformation methods are also supported. Fourth, it provides robustness: strong validation of field migration completeness and the total number of entries before and after migration. Fifth, it enhances code reusability: common elements are easily extracted. For business users, it offers the following benefits: First, the data migration process is safer and more seamless, with the new system well compatible with the old data in the old system. Second, the migration process is faster, shortening the time for parallel operation of the new and old systems. Third, it improves the system's running speed and facilitates the development of new business features.

[0085] This application, through annotations, is not limited to data migration between two databases, but also applicable to data migration within different storage spaces of the same database. Another embodiment of this application provides a data processing method, including the following steps:

[0086] S801, Obtain the first entity class of the first data table to be migrated in the migration storage space and the second entity class of the second data table in the migration storage space; wherein, the migration storage space and the migration storage space are located in the same database;

[0087] S802, generate at least one first annotation based on the operation information of the first data table, wherein the operation information is used to indicate the content of the field to be changed in the second data table relative to the first data table, each first annotation corresponds to a second field in the second data table, each first annotation includes an operation type code and a first field in the first data table, and the data of the second field is obtained based on the data of the first field;

[0088] S803, a migration task is created based on the first entity class, the second entity class, and the at least one first annotation. The migration task is used to migrate the data of the first data table in the migrated storage space to the second data table in the migration storage space.

[0089] Accordingly, after creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the process further includes:

[0090] At least one second annotation is generated based on the verification information of the second data table, wherein each second annotation includes a verification type code;

[0091] Add the at least one second annotation to the migration task.

[0092] Accordingly, after creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the process further includes:

[0093] At least one third annotation is generated based on the foreign key association information of the second data table, wherein each third annotation includes foreign key migration code, a third field in the first data table that serves as the primary key, and a fourth field that serves as the foreign key and is associated with the third field.

[0094] Retrieve the first annotation corresponding to the fourth field;

[0095] If the acquisition is successful, the third annotation is added to the first annotation of the migration task corresponding to the fourth field;

[0096] If the retrieval fails, a corresponding first annotation is generated based on the second field corresponding to the fourth field in the second data table and the third annotation.

[0097] like Figure 4As shown, one embodiment of this application provides a data processing apparatus 40, which includes a creation module 41, an annotation module 42, and a task module 43. The creation module 41 is used to obtain a first entity class of a first data table to be migrated in a migrated database and a second entity class of a second data table in a migration database. The annotation module 42 is used to generate at least one first annotation based on operation information of the first data table, wherein the operation information indicates the content of a field to be changed in the second data table relative to the first data table, each first annotation corresponds to a second field in the second data table, and each first annotation includes an operation type code and a first field in the first data table, the data of the second field being obtained based on the data of the first field. The task module 43 is used to create a migration task based on the first entity class, the second entity class, and the at least one first annotation, the migration task being used to migrate the data of the first data table in the migrated database to the second data table in the migration database.

[0098] In some optional implementations, the annotation module 42 is further configured to: generate at least one second annotation based on the verification information of the second data table, wherein each second annotation includes a verification type code; and add the at least one second annotation to the migration task.

[0099] In some optional implementations, the annotation module 42 is further configured to: generate at least one third annotation based on the foreign key association information of the second data table, wherein each third annotation includes foreign key migration code, a third field in the first data table that serves as the primary key, and a fourth field associated with the third field that serves as the foreign key; obtain the first annotation corresponding to the fourth field; if the acquisition is successful, add the third annotation to the first annotation corresponding to the fourth field in the migration task; if the acquisition fails, generate a corresponding first annotation based on the second field in the second data table that corresponds to the fourth field and the third annotation.

[0100] In some alternative implementations, the data processing apparatus further includes an execution module for performing the migration task to migrate data from the first data table to the second data table.

[0101] In some optional implementations, the execution module is further configured to: determine whether the current field of the first data table is a field to be changed; when the current field is determined to be a field to be changed, perform the processing corresponding to the operation type code on the data of the current field according to the first annotation corresponding to the current field, and migrate the processed data to the second field of the second data table; when the current field is determined not to be a field to be changed, migrate the data of the current field to the second data table.

[0102] In some optional implementations, after determining that the current field is a field to be changed, performing the processing corresponding to the operation type code on the data of the current field according to the first annotation corresponding to the current field, and migrating the processed data to the second field of the second data table, the execution module further includes: according to the third annotation, associating the fourth field as a foreign key with the second field as a primary key, and removing the primary and foreign key association relationship between the fourth field and the corresponding field of the third field in the second data table.

[0103] In some optional implementations, after determining that the current field is a field to be changed, performing the processing corresponding to the operation type code on the data of the current field according to the first annotation corresponding to the current field, and migrating the processed data to the second field of the second data table, the execution module further includes: performing the processing corresponding to the verification type code on the data of all fields of the second data table according to the second annotation.

[0104] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Figure 5 As shown, the electronic device 50 includes a processor 51 and a memory 52 coupled to the processor 51.

[0105] The memory 52 stores program instructions for implementing the data processing method of any of the above embodiments.

[0106] The processor 51 is used to execute program instructions stored in the memory 52 for data processing.

[0107] The processor 51 can also be referred to as a CPU (Central Processing Unit). The processor 51 may be an integrated circuit chip with signal processing capabilities. The processor 51 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor.

[0108] See Figure 6 , Figure 6 This is a schematic diagram of the structure of a storage medium according to an embodiment of this application. The storage medium 60 of this embodiment stores program instructions 61 capable of implementing all the methods described above. These program instructions 61 can be stored in the storage medium in the form of a software product, including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks, or terminal devices such as computers, servers, mobile phones, and tablets.

[0109] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0110] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0111] The above description is merely an embodiment of this application. It should be noted that those skilled in the art can make improvements without departing from the inventive concept of this application, but these improvements all fall within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Retrieve the first entity class of the first data table to be migrated in the database being migrated, and the second entity class of the second data table in the migration database; At least one first annotation is generated based on the operation information of the first data table, wherein the operation information is used to indicate the content of the field to be changed in the second data table relative to the first data table, each first annotation corresponds to a second field in the second data table, and each first annotation includes an operation type code and a first field in the first data table, wherein the data of the second field is obtained based on the data of the first field; A migration task is created based on the first entity class, the second entity class, and the at least one first annotation. The migration task is used to migrate data from the first data table in the migrated database to the second data table in the migration database.

2. The data processing method according to claim 1, characterized in that, After creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the process further includes: At least one second annotation is generated based on the verification information of the second data table, wherein each second annotation includes a verification type code; Add the at least one second annotation to the migration task.

3. The data processing method according to claim 1, characterized in that, After creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the process further includes: At least one third annotation is generated based on the foreign key association information of the second data table, wherein each third annotation includes foreign key migration code, a third field in the first data table that serves as the primary key, and a fourth field that serves as the foreign key and is associated with the third field. Retrieve the first annotation corresponding to the fourth field; If the acquisition is successful, the third annotation is added to the first annotation of the migration task corresponding to the fourth field; If the retrieval fails, a corresponding first annotation is generated based on the second field corresponding to the fourth field in the second data table and the third annotation.

4. The data processing method according to claim 1, characterized in that, After creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the process further includes: The migration task is executed to migrate data from the first data table to the second data table.

5. The data processing method according to claim 4, characterized in that, Executing the migration task to migrate data from the first data table to the second data table includes: Determine whether the current field in the first data table is the field to be changed; When it is determined that the current field is a field to be changed, according to the first annotation corresponding to the current field, the data of the current field is processed according to the operation type code, and the processed data is migrated to the second field of the second data table; When it is determined that the current field is not a field to be changed, the data of the current field is migrated to the second data table.

6. The data processing method according to claim 5, characterized in that, After creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the process further includes: At least one third annotation is generated based on the foreign key association information of the second data table, wherein each third annotation includes foreign key migration code, a third field in the first data table that serves as the primary key, and a fourth field that serves as the foreign key and is associated with the third field. Retrieve the first annotation corresponding to the fourth field; If the acquisition is successful, the third annotation is added to the first annotation of the migration task corresponding to the fourth field; If the retrieval fails, the corresponding first annotation is generated based on the second field corresponding to the fourth field in the second data table and the third annotation; Accordingly, after determining that the current field is a field to be changed, performing the processing corresponding to the operation type code on the data of the current field according to the first annotation corresponding to the current field, and migrating the processed data to the second field of the second data table, the method further includes: According to the third annotation, the fourth field is associated with the second field as a primary key, thereby removing the primary and foreign key association between the fourth field and the corresponding field of the third field in the second data table.

7. The data processing method according to claim 5, characterized in that, After creating the migration task based on the first entity class, the second entity class, and the at least one annotation, the process further includes: At least one second annotation is generated based on the verification information of the second data table, wherein each second annotation includes a verification type code; Add the at least one second annotation to the migration task; Accordingly, after determining that the current field is a field to be changed, performing the processing corresponding to the operation type code on the data of the current field according to the first annotation corresponding to the current field, and migrating the processed data to the second field of the second data table, the method further includes: Based on the second annotation, perform the processing corresponding to the validation type code on the data of all fields in the second data table.

8. A data processing apparatus, characterized in that, include: Create a module to obtain the first entity class of the first data table to be migrated in the database being migrated, and the second entity class of the second data table in the migration database; An annotation module is used to generate at least one first annotation based on the operation information of the first data table, wherein the operation information is used to indicate the content of the field to be changed in the second data table relative to the first data table, each first annotation corresponds to a second field in the second data table, and each first annotation includes an operation type code and a first field in the first data table, wherein the data of the second field is obtained based on the data of the first field; The task module is used to create a migration task based on the first entity class, the second entity class, and the at least one first annotation. The migration task is used to migrate data from the first data table in the migrated database to the second data table in the migration database.

9. An electronic device, characterized in that, The method includes a processor and a memory coupled to the processor, the memory storing program instructions executable by the processor; when the processor executes the program instructions stored in the memory, it implements the data processing method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores program instructions, which, when executed by a processor, implement the data processing method as described in any one of claims 1 to 7.

Images