A data migration method, system, computer device and readable medium

Abstract

This application provides a data migration method, system, computer equipment, and readable medium. The method involves acquiring the data to be migrated and performing business logic layer matching to determine the business logic layer of the data. A proxy persistence layer is added between the business logic layer and a preset persistence layer. A business switch in the proxy persistence layer is used to configure switch values ​​for the data to be migrated. Based on the configured switch values, a data migration statement is selected, and the selected statement is used to migrate the data from a first target database to a second target database, enabling a persistent switching between the first and second target databases. This application, by adding a proxy persistence layer, achieves logical segmentation at the code level without modifying the original logic code. Furthermore, because no changes are made to the existing code, security is ensured, and data migration costs are reduced. This method is suitable for code-free transformation of most business systems.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data migration method, system, computer device, and readable medium. Background Technology

[0002] MySQL is a relational database management system. Relational databases store data in different tables, rather than putting all the data in one large repository, which increases speed and improves flexibility. Oracle is currently the most widely used database management system in the world. As a general-purpose database system, it has complete data management functions; as a relational database, it is a complete relational product; and as a distributed database, it implements distributed processing capabilities.

[0003] MySQL and Oracle are commonly used database management systems in enterprise business systems, but they are not fully compatible. Therefore, for data persistence purposes, it may be necessary to migrate the business system's database from Oracle to MySQL. However, current data migration methods involve modifying existing logic code, resulting in high migration costs. Furthermore, existing migration methods introduce redundant code. Additionally, the different data field types between Oracle and MySQL databases can cause data discrepancies before and after migration, making asynchronous processing unsupported after the migration. Summary of the Invention

[0004] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a data migration method, system, computer device and readable medium to solve the technical problems existing in the prior art.

[0005] To achieve the above and other related objectives, this application provides a data migration method, which includes the following steps:

[0006] Obtain the data to be migrated, and perform business logic layer matching based on the business logic type of the data to be migrated to determine the business logic layer of the data to be migrated;

[0007] A proxy persistence layer is added between the business logic layer and the preset persistence layer, and the business switches in the proxy persistence layer are used to configure the switch values ​​of the data to be migrated.

[0008] Based on the configured switch value, a data migration statement is selected, and the selected data migration statement is used to migrate the data to be migrated from the first target database to the second target database, so that the data to be migrated is persistently switched between the first target database and the second target database.

[0009] In one embodiment of this application, the process of adding a proxy persistence layer between the business logic layer and the preset persistence layer includes:

[0010] The logic interface is created by creating a pre-defined object to simulate the generation of access objects in the target container framework, and the simulated access objects are then transmitted to the pre-defined container.

[0011] The access objects in the preset container are dynamically managed so that the access objects in the preset container are logically enhanced after dynamic management;

[0012] Obtain parameters for different target types, and instantiate the dynamically managed access objects according to the preset construction method. Based on the instantiation result, add a proxy persistence layer between the business logic layer and the preset persistence layer.

[0013] In one embodiment of this application, the process of configuring the switch value of the data to be migrated using the business switch in the proxy persistence layer includes: adding a business switch to the proxy persistence layer, configuring the switch value of the data to be migrated using the business switch, and selecting a switch value corresponding to the data to be migrated; wherein, the switch value used for selection includes at least one of the following: oracle, mysql, all.

[0014] In one embodiment of this application, the process of selecting a data migration statement based on a configured switch value and using the selected data migration statement to migrate the data to be migrated from a first target database to a second target database includes:

[0015] Obtain the configured switch value for the number to be migrated, and record it as the target switch value;

[0016] Select a data migration statement associated with the target switch value, and use the selected data migration statement to migrate the data to be migrated from the first target database to the second target database; or,

[0017] Select a data migration statement associated with the target switch value, and asynchronously perform a comparison of the data to be migrated in the first target database and the second target database using the selected data migration statement.

[0018] In one embodiment of this application, the first target database is an Oracle database and the second target database is a MySQL database; or, the first target database is a MySQL database and the second target database is an Oracle database.

[0019] In one embodiment of this application, the process of obtaining the data to be migrated includes:

[0020] The system retrieves the corresponding table name information by calling all database tables in the MySQL database using a preset display table statement.

[0021] By using a preset statement to iterate through all database tables, the initial table structure definition information is obtained. The initial table structure definition information includes field column names, field column data types, default values, and additional information.

[0022] The database table and the initial table structure definition information are concatenated and transformed to generate the data to be migrated.

[0023] In one embodiment of this application, after migrating the data to be migrated from the first target database to the second target database using the selected data migration statement, the method further includes:

[0024] Obtain the data that has been migrated to the second target database, and denote it as the target migration data;

[0025] Extract keywords or key fields from the target migration data, and segment the target migration data based on the keywords or key fields.

[0026] This application also provides a data migration system, the system comprising:

[0027] The data acquisition module is used to acquire the data to be migrated.

[0028] The data matching module is used to perform business logic layer matching based on the business logic type of the data to be migrated, and to determine the business logic layer of the data to be migrated.

[0029] A switch value configuration module is used to add a proxy persistence layer between the business logic layer and the preset persistence layer, and to configure the switch values ​​of the data to be migrated using the business switches in the proxy persistence layer.

[0030] The data migration module is used to select a data migration statement according to the configured switch value, and use the selected data migration statement to migrate the data to be migrated from the first target database to the second target database, so that the data to be migrated can be persistently switched between the first target database and the second target database.

[0031] This application also provides a computer device, the computer device comprising:

[0032] One or more processors;

[0033] A storage device for storing one or more programs, which, when executed by one or more processors, cause the electronic device to perform the data migration method as described in any of the preceding descriptions.

[0034] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer's processor, causes the computer to perform the data migration method as described in any of the above-described methods.

[0035] As described above, this application provides a data migration method, system, computer device, and readable medium, which have the following beneficial effects:

[0036] This application first acquires the data to be migrated, then matches the business logic layer based on the business logic type of the data to be migrated to determine the business logic layer of the data to be migrated; next, a proxy persistence layer is added between the business logic layer and the preset persistence layer, and the business switches in the proxy persistence layer are used to configure the switch values ​​of the data to be migrated; finally, a data migration statement is selected based on the configured switch values, and the selected data migration statement is used to migrate the data to be migrated from the first target database to the second target database, so that the data to be migrated can be persistently switched between the first target database and the second target database. Therefore, this application ensures code-free modification by embedding a proxy persistence layer. It achieves zero modification to the original logic code by adding a proxy layer between the original business logic layer and the persistence layer, forming a proxy persistence layer. This not only achieves logical separation at the code level but is also applicable to most business systems for code-free modification, ensuring the security and reliability of the data migration process. In other words, this application, by embedding a proxy persistence layer, not only achieves zero modification to the existing code, ensuring security and reliability, but also reduces data migration costs; furthermore, this application supports switching between on / off data persistence methods, enabling online switching and dual-write asynchronous verification. Furthermore, this application involves lightweight modifications throughout the entire data migration process, making it largely imperceptible to users and thus applicable to most business systems for code-free transformation. Essentially, this application, based on the idea of ​​zero modification to the original logic code, adds a proxy persistence layer, achieving not only logical separation at the code level but also ensuring security and reducing data migration costs because no changes are made to the existing code. This makes it suitable for most business systems for code-free transformation. Attached Figure Description

[0037] Figure 1 This is a schematic diagram illustrating an exemplary system architecture that applies the technical solutions in one or more embodiments of this application;

[0038] Figure 2 A schematic flowchart illustrating a data migration method provided in an embodiment of this application;

[0039] Figure 3 A schematic flowchart illustrating a data migration method provided in another embodiment of this application;

[0040] Figure 4 This is a schematic diagram of the hardware structure of a data migration system provided in an embodiment of this application;

[0041] Figure 5 This is a schematic diagram of the hardware structure of a computer device suitable for implementing one or more embodiments of this application. Detailed Implementation

[0042] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0043] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0044] Oracle: The full English name is Oracle Database, also known as Oracle RDBMS, or simply Oracle. Oracle Database System is a suite of software products provided by Oracle Corporation, an American company. It is a distributed database and one of the most popular client / server (CLIENT / SERVER) or browser / server (B / S) architecture databases. For example, SilverStream is a middleware based on this database. Oracle Database is the world's most widely used database management system. As a general-purpose database system, it has complete data management functions; as a relational database, it is a complete relational product; and as a distributed database, it implements distributed processing capabilities. However, once you learn Oracle on one type of machine, you can use it on various types of machines.

[0045] MySQL: It is a relational database management system developed by the Swedish company MySQL AB and is a product of Oracle. MySQL is one of the most popular relational database management systems, and in web applications, it is one of the best RDBMS (Relational Database Management System) applications. MySQL is a relational database management system; relational databases store data in different tables, rather than putting all the data in one large repository, thus increasing speed and improving flexibility.

[0046] SQL stands for Structured Query Language. It's a database language with multiple functions, including data manipulation and definition. This interactive language provides great convenience to users, and database management systems should fully utilize SQL to improve the quality and efficiency of computer application systems. SQL can be used independently in terminals or as a sub-language to effectively assist in other program design. In application, SQL can work with other programming languages ​​to optimize program functionality, thereby providing users with more comprehensive information. The SQL Server database includes two sub-databases: Microsoft SQL Server and Sybase SQL Server. The proper functioning of this database directly affects the operational security of the entire computer system.

[0047] DAO: Data Access Object, a proxy persistence layer. The DAO layer acts as an intermediary between the business logic layer and the database layer, significantly reducing development costs and improving system maintenance efficiency. Business logic code interacts with the database by calling DAO components.

[0048] FactoryBean is a pattern that encapsulates the logic for creating a target object within a single class. For example, it can be used to create objects in complex object graphs in a reusable coding manner. It is typically used to build complex objects with many dependencies. It can also be used to solve problems where the construction logic is not fixed and depends on configuration to create objects. FactoryBean can also help Spring create objects that it cannot easily create itself. For example, in order to inject a reference into a bean obtained from JNDI, the reference must first be obtained.

[0049] Spring is a lightweight open-source framework in the Java EE programming field. It was first proposed and subsequently created by a programmer named Rod Johnson in 2002. It's an application framework designed to address the complexities of enterprise-level programming development and enable agile development. Spring is an open-source container framework that integrates various types of tools. Through its core Bean Factory, it implements the underlying instantiation and lifecycle management of classes. Within the framework, various functionalities are abstracted into individual Beans, enabling the management of various functionalities, including dynamic loading and aspect-oriented programming.

[0050] Dynamic proxy: The specific method for generating objects, that is, the assembly manual. The difference is that the manual is fixed, while the proxy can be customized at will.

[0051] Figure 1 A schematic diagram of an exemplary system architecture that can apply the technical solutions of one or more embodiments of this application is shown. Figure 1 As shown, the system architecture 100 may include terminal device 110, network 120, and server 130. Terminal device 110 may include various electronic devices such as smartphones, tablets, laptops, and desktop computers. Server 130 may be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Network 120 may be a communication medium of various connection types capable of providing a communication link between terminal device 110 and server 130, such as a wired communication link or a wireless communication link.

[0052] Depending on the implementation requirements, the system architecture in this application embodiment can have any number of terminal devices, networks, and servers. For example, server 130 can be a server group composed of multiple server devices. In addition, the technical solutions provided in this application embodiment can be applied to terminal device 110, or to server 130, or can be implemented jointly by terminal device 110 and server 130. This application does not impose any special limitations on this.

[0053] In one embodiment of this application, the terminal device 110 or server 130 can first obtain the data to be migrated, then perform business logic layer matching based on the business logic type of the data to be migrated to determine the business logic layer of the data to be migrated; then, add a proxy persistence layer between the business logic layer and the preset persistence layer, and use the business switches in the proxy persistence layer to configure the switch values ​​of the data to be migrated; finally, select a data migration statement based on the configured switch values, and use the selected data migration statement to migrate the data to be migrated from the first target database to the second target database, so that the data to be migrated can be persistently switched between the first target database and the second target database. By using the terminal device 110 or server 130 to execute the data migration method, the mechanism of embedding a proxy persistence layer can ensure code removal. Based on the idea of ​​zero modification to the original logic code, by adding a proxy layer between the original business logic layer and the persistence layer, a proxy persistence layer is formed. This not only achieves logical segmentation at the code level, but is also applicable to most business systems for code removal, ensuring the security and reliability of the data migration process. Specifically, by embedding a proxy persistence layer during the data migration process using terminal device 110 or server 130, not only is zero modification to existing code achieved, ensuring security and reliability, but data migration costs are also reduced. Furthermore, it supports switching between on / off data persistence methods, enabling online switching, and supports dual-write asynchronous verification. Moreover, the entire data migration process is lightweight and imperceptible to users, making it suitable for most business systems requiring no code modification.

[0054] The above sections introduced an exemplary system architecture that applies the technical solution of this application. Next, we will continue to introduce the data migration method of this application.

[0055] Figure 2 A schematic flowchart of a data migration method provided in an embodiment of this application is shown. Specifically, in an exemplary embodiment, as follows... Figure 2 As shown, this embodiment provides a data migration method, which includes the following steps:

[0056] S210, Obtain the data to be migrated, and perform business logic layer matching based on the business logic type of the data to be migrated to determine the business logic layer of the data to be migrated;

[0057] S220, a proxy persistence layer is added between the business logic layer and the preset persistence layer, and the business switch in the proxy persistence layer is used to configure the switch value of the data to be migrated.

[0058] S230: Based on the configured switch value, a data migration statement is selected, and the selected data migration statement is used to migrate the data to be migrated from the first target database to the second target database, so that the data to be migrated is persistently switched between the first target database and the second target database. As an example, in this embodiment, the first target database is an Oracle database and the second target database is a MySQL database; as another example, in this embodiment, the first target database is a MySQL database and the second target database is an Oracle database.

[0059] Therefore, this embodiment ensures code-free transformation by embedding a proxy persistence layer. It achieves zero modification to the original logic code by adding a proxy layer between the original business logic layer and the persistence layer, forming a proxy persistence layer. This not only achieves logical separation at the code level but is also applicable to most business systems for code-free transformation, ensuring the security and reliability of the data migration process. In other words, this embodiment, by embedding a proxy persistence layer, not only achieves zero modification to existing code, ensuring security and reliability, but also reduces data migration costs. Furthermore, this embodiment supports switching between on / off data persistence methods, enabling online switching and supporting asynchronous dual-write verification. In addition, this embodiment involves lightweight modifications throughout the entire data migration process, which are not easily perceived by users, making it applicable to most business systems for code-free transformation. If the first target database is a MySQL database and the second target database is an Oracle database, this embodiment can migrate data from a MySQL database to an Oracle database. If the first target database is an Oracle database and the second target database is a MySQL database, this embodiment can migrate data from an Oracle database to a MySQL database.

[0060] In an exemplary embodiment, the process of adding a proxy persistence layer between the business logic layer and the preset persistence layer includes: simulating the generation of access objects in the target container framework through a preset object creation logic interface, and transmitting the simulated access objects to the preset container; dynamically managing the access objects in the preset container to enhance their logic after dynamic management; obtaining parameters of different target types, and instantiating the dynamically managed access objects according to a preset construction method; and adding a proxy persistence layer between the business logic layer and the preset persistence layer based on the instantiation result. Specifically, in this embodiment, an object bean in the target container framework Spring is simulated through the object creation logic interface FactoryBean, and then the simulated object bean is transmitted to the preset container IoC for management. The access objects in the preset container are then dynamically managed to enhance their logic before and after dynamic management. Finally, parameters of different target types (e.g., Class type) are accepted through generics, and the dynamically managed access objects are instantiated using a preset constructor method to create specific object beans. Finally, based on the instantiated Bean object, a proxy persistence layer is added between the business logic layer and the preset persistence layer.

[0061] In an exemplary embodiment, the process of configuring a switch value for the data to be migrated using a business switch in the proxy persistence layer includes: adding a business switch to the proxy persistence layer, configuring a switch value for the data to be migrated using the business switch, and selecting a switch value corresponding to the data to be migrated; wherein the switch value used for selection includes at least one of the following: oracle, mysql, all. Specifically, the process of selecting a data migration statement based on the configured switch value and using the selected data migration statement to migrate the data to be migrated from a first target database to a second target database includes: obtaining the switch value configured for the data to be migrated, denoted as the target switch value; selecting a data migration statement associated with the target switch value, and using the selected data migration statement to migrate the data to be migrated from the first target database to the second target database; or, selecting a data migration statement associated with the target switch value, and using the selected data migration statement to asynchronously perform a content comparison of the data to be migrated in the first target database and the second target database. As an example, if the configured switch value for the data to be migrated is "oracle", then the data migration statement associated with the "oracle" switch value is selected, and the data to be migrated is migrated from the first target database to the second target database using the data migration statement associated with the "oracle" switch value. As another example, if the configured switch value for the data to be migrated is "mysql", then the data migration statement associated with the "mysql" switch value is selected, and the data to be migrated is migrated from the first target database to the second target database using the data migration statement associated with the "mysql" switch value. As yet another example, if the configured switch value for the data to be migrated is "all", then the data migration statements associated with the "oracle" and "mysql" switch values ​​are selected, and the content comparison of the data to be migrated in the first target database and the second target database is performed asynchronously using the data migration statements associated with the "oracle" and "mysql" switch values.

[0062] In an exemplary embodiment, the process of obtaining the data to be migrated includes: calling all database tables in the MySQL database using a preset show table statement to obtain the corresponding table name information; iterating through all database tables using a preset show table statement to obtain initial table structure definition information, which includes field column names, data types of the field columns, default values, and additional information; and concatenating and converting the database tables and the initial table structure definition information to generate the data to be migrated. Specifically, the table name information of all database tables in the initial MySQL database is obtained. In this embodiment, a data table consists of three parts: table name, fields in the table, and records in the table. Designing the data table structure involves defining the data table name, determining which fields the data table contains, and the field names, field types, and widths of each field. This solution obtains the table name information of all database tables in the initial MySQL database, and can further obtain the initial table structure definition information of the corresponding data table based on the table name. The show table statement calls all database tables in the initial MySQL database to obtain the corresponding table name information. The show table statement can display all data tables in the current initial MySQL database, and the corresponding table name information can be directly obtained based on the displayed data tables. Then, the `desc + [table name]` statement is used to iterate through the corresponding database tables to obtain the initial table structure definition information. The `desc + [table name]` statement displays which fields are defined in the data table, the type and size of each field, which are primary keys, which have constraints, and whether each field has a default value, etc. This is equivalent to writing the `show table` and `desc + [table name]` statements as a Python script and importing it into the terminal. The script automatically connects to the MySQL database and executes database definition statements like `show tables` and `desc + [table name]` to obtain all table name information and table structure definition information. The database tables and the initial table structure definition information are then concatenated and transformed to generate the data to be migrated. In this embodiment, MySQL and Oracle, as two mainstream relational database systems, differ slightly in their usage. Although they are compatible with most SQL query statements, there are still some areas where they are not fully compatible. Since MySQL and Oracle are not fully compatible, the obtained database table and initial table structure definition information needs to be concatenated and transformed using corresponding conversion rules. This ensures that the data to be migrated conforms to the target table structure definition information required by the Oracle or MySQL database, thereby guaranteeing the integrity of the migrated database and improving data security.

[0063] In an exemplary embodiment, after migrating the data to be migrated from a first target database to a second target database using the selected data migration statement, the method further includes: obtaining the data migrated to the second target database, denoted as target migration data; extracting keywords or key fields from the target migration data, and sharding the target migration data based on the keywords or key fields. In this embodiment, the keywords or key fields in the target migration data include shard refresh parameters, which include: shard size, page size, and total data volume. Specifically, firstly, the start index and end index corresponding to the first page number in the database are determined according to the first page number and the page size. Then, the start index of the first page number in each database shard is determined according to the start index, page size, and total data volume. Finally, the end index of the first page number in each database shard is determined according to the end index, page size, and total data volume. Then, according to the start index and end index in each database shard, the corresponding data values ​​are obtained from each sorted database shard, and the data values ​​are combined to form a first intermediate data set. Specifically, in this embodiment, the first page number is defined as `pageIndex`, the database shard size is `sliceSize`, the page size is `pageSize`, and the total data volume is the total number of data stored in the database, defined as `total`. From the database configuration in the above scheme, the first page number `pageIndex` is 3, the database shard size `sliceSize` is 25, the page size `pageSize` is 4, and the total data volume `total` is 100. First, based on the first page number `pageIndex` and the page size `pageSize`, the starting index `startIndex` and the ending index `endIndex` corresponding to the first page number `pageIndex` in all data in the database are calculated, where: `startIndex = pageSize * (pageIndex - 1)`; `endIndex = pageSize * pageIndex - 1`. Since the first page number pageIndex is 3 and pageSize is 4, therefore: startIndex = 4 * (3 - 1) = 8; endIndex = 4 * 3 - 1 = 11; then the starting index of the third page corresponding to the first page number in the database is 8 and the ending index of the page is 11 in the sorted database.Then, based on the start index startIndex, end index endIndex, slice size sliceSize, and total data volume corresponding to the first page number, calculate the start index startIndexInSlice and end index endIndexInSlice of the first page number in each database slice, where: startIndexInSlice = sliceSize * startIndex / total; endIndexInSlice = sliceSize * endIndex / total; and startIndexInSlice is the value after rounding down, and endIndexInSlice is the value after rounding up. Since sliceSize is 25, total is 100, startIndex is 8, and endIndex is 11, therefore: startIndexInSlice = 25 * 8 / 100 = 2; endIndexInSlice = 25 * 11 / 100 = 2.75; after rounding endIndexInSlice up, endIndexInSlice is 3. Finally, according to the start and end indices in the database sharding, the corresponding data values ​​are obtained from each database shard, and the data values ​​form the first intermediate data set, completing the sharding of the target migration data.

[0064] In another exemplary embodiment of this application, such as Figure 3 As shown, this embodiment also provides a data migration method, including the following steps:

[0065] The data to be migrated is obtained and verified by the BIZ layer before being transmitted to the business editing layer. The BIZ layer is used to perform rule verification, such as ensuring that user passwords are correct and not empty, that IP addresses and usernames are not frozen, and that users are not added to blacklists.

[0066] Between the business editing layer SERVICE and the persistence layer DAO, a logical proxy_dao is added as a persistence proxy layer. As an example, this embodiment simulates the generation of object beans in the target container framework Spring through the object creation logic interface FactoryBean, and then transmits the simulated object beans to a preset container IoC for management. The access objects in the preset container are then dynamically managed to enhance their logic before and after dynamic management. Generics are used to accept parameters of different target types (e.g., Class type), and a preset constructor is used to instantiate the dynamically managed access objects, creating specific object beans. Finally, based on the instantiated object beans, a proxy persistence layer proxy_dao is added between the business editing layer SERVICE and the persistence layer DAO. In this embodiment, the persistence layer DAO includes the MYSQL_DAO layer and the ORACLE_DAO layer.

[0067] Next, configure a switch OPS for the proxy persistence layer proxy_dao, and use the configured switch OPS to determine the database type dbType of the data to be migrated, and configure the switch value for the data to be migrated. The switch value used for selection includes at least one of the following: oracle, mysql, or all.

[0068] Based on the configured switch values, data migration statements are selected and used to migrate the data from the MySQL database to the Oracle database, enabling persistent switching between the two databases. Specifically, if the configured switch value for the data to be migrated is "oracle", the data migration statement associated with that value is selected and used to migrate the data from the first target database to the second target database. If the configured switch value is "mysql", the data migration statement associated with that value is selected and used to migrate the data from the first target database to the second target database. If the configured switch value is "all", the data migration statements associated with both "oracle" and "mysql" values ​​are selected and used to asynchronously compare the contents of the data to be migrated between the first and second target databases.

[0069] Based on the above description, as an example, when a company needs to recruit due to business expansion, it can first open a recruitment window and define the talent requirements, creating a proxy_dao. During specific business operations, different levels are set according to the specific circumstances of the interviewees, and the value of the OPS switch output is obtained. If the OPS switch output value is oracle, the data is saved to the Oracle database; if the OPS switch output value is mysql, the data is saved to the MySQL database; if the switch value is all, both the Oracle and MySQL databases retain a copy, with the data in the Oracle database serving as the actual data and the data in the MySQL database serving as the comparison data. This allows for dual-write asynchronous verification, asynchronously comparing the content of the two sides of the data.

[0070] In summary, this application provides a data migration method. First, it acquires the data to be migrated. Then, it matches the business logic layer based on the business logic type of the data to be migrated to determine the business logic layer. Next, it adds a proxy persistence layer between the business logic layer and a preset persistence layer, and uses business switches in the proxy persistence layer to configure switch values ​​for the data to be migrated. Finally, it selects a data migration statement based on the configured switch values ​​and uses the selected statement to migrate the data from a first target database to a second target database, thus enabling a persistent switching between the first and second target databases. Therefore, this method ensures code-free modification by embedding a proxy persistence layer. It achieves logical separation at the code level by adding a proxy layer between the original business logic layer and the persistence layer, without requiring any modification to the original logic code. This is applicable to most business systems for code-free modification and ensures the security and reliability of the data migration process. This method, by embedding a proxy persistence layer, not only achieves zero modification to existing code, ensuring security and reliability, but also reduces data migration costs. Furthermore, it supports switching between on / off data persistence methods, enabling online switching and dual-write asynchronous verification. In addition, the entire data migration process is lightweight and imperceptible to users, making it suitable for most business systems requiring no code modification. Essentially, this method, based on the idea of ​​zero modification to existing logic code, adds a proxy persistence layer to achieve logical separation at the code level. Because it does not modify existing code, it ensures security and reduces data migration costs, making it suitable for most business systems requiring no code modification. If the first target database is a MySQL database and the second target database is an Oracle database, this method can migrate data from a MySQL database to an Oracle database. Conversely, if the first target database is an Oracle database and the second target database is a MySQL database, this method can migrate data from an Oracle database to a MySQL database.

[0071] like Figure 4 As shown, one embodiment of this application also provides a data migration system, which includes:

[0072] Data acquisition module 410 is used to acquire the data to be migrated;

[0073] Data matching module 420 is used to perform business logic layer matching based on the business logic type of the data to be migrated, and to determine the business logic layer of the data to be migrated.

[0074] The switch value configuration module 430 is used to add a proxy persistence layer between the business logic layer and the preset persistence layer, and to configure the switch value of the data to be migrated using the business switches in the proxy persistence layer.

[0075] The data migration module 440 is used to select a data migration statement according to the configured switch value, and use the selected data migration statement to migrate the data to be migrated from the first target database to the second target database, so that the data to be migrated can be persistently switched between the first target database and the second target database. As an example, in this embodiment, the first target database is an Oracle database and the second target database is a MySQL database; as another example, in this embodiment, the first target database is a MySQL database and the second target database is an Oracle database.

[0076] Therefore, this embodiment ensures code-free transformation by embedding a proxy persistence layer. It achieves zero modification to the original logic code by adding a proxy layer between the original business logic layer and the persistence layer, forming a proxy persistence layer. This not only achieves logical separation at the code level but is also applicable to most business systems for code-free transformation, ensuring the security and reliability of the data migration process. In other words, this embodiment, by embedding a proxy persistence layer, not only achieves zero modification to existing code, ensuring security and reliability, but also reduces data migration costs. Furthermore, this embodiment supports switching between on / off data persistence methods, enabling online switching and supporting asynchronous dual-write verification. In addition, this embodiment involves lightweight modifications throughout the entire data migration process, which are not easily perceived by users, making it applicable to most business systems for code-free transformation. If the first target database is a MySQL database and the second target database is an Oracle database, this embodiment can migrate data from a MySQL database to an Oracle database. If the first target database is an Oracle database and the second target database is a MySQL database, this embodiment can migrate data from an Oracle database to a MySQL database.

[0077] In an exemplary embodiment, the process of adding a proxy persistence layer between the business logic layer and the preset persistence layer includes: simulating the generation of access objects in the target container framework through a preset object creation logic interface, and transmitting the simulated access objects to the preset container; dynamically managing the access objects in the preset container to enhance their logic after dynamic management; obtaining parameters of different target types, and instantiating the dynamically managed access objects according to a preset construction method; and adding a proxy persistence layer between the business logic layer and the preset persistence layer based on the instantiation result. Specifically, in this embodiment, an object bean in the target container framework Spring is simulated through the object creation logic interface FactoryBean, and then the simulated object bean is transmitted to the preset container IoC for management. The access objects in the preset container are then dynamically managed to enhance their logic before and after dynamic management. Finally, parameters of different target types (e.g., Class type) are accepted through generics, and the dynamically managed access objects are instantiated using a preset constructor method to create specific object beans. Finally, based on the instantiated Bean object, a proxy persistence layer is added between the business logic layer and the preset persistence layer.

[0078] In an exemplary embodiment, the process of configuring a switch value for the data to be migrated using a business switch in the proxy persistence layer includes: adding a business switch to the proxy persistence layer, configuring a switch value for the data to be migrated using the business switch, and selecting a switch value corresponding to the data to be migrated; wherein the switch value used for selection includes at least one of the following: oracle, mysql, all. Specifically, the process of selecting a data migration statement based on the configured switch value and using the selected data migration statement to migrate the data to be migrated from a first target database to a second target database includes: obtaining the switch value configured for the data to be migrated, denoted as the target switch value; selecting a data migration statement associated with the target switch value, and using the selected data migration statement to migrate the data to be migrated from the first target database to the second target database; or, selecting a data migration statement associated with the target switch value, and using the selected data migration statement to asynchronously perform a content comparison of the data to be migrated in the first target database and the second target database. As an example, if the configured switch value for the data to be migrated is "oracle", then the data migration statement associated with the "oracle" switch value is selected, and the data to be migrated is migrated from the first target database to the second target database using the data migration statement associated with the "oracle" switch value. As another example, if the configured switch value for the data to be migrated is "mysql", then the data migration statement associated with the "mysql" switch value is selected, and the data to be migrated is migrated from the first target database to the second target database using the data migration statement associated with the "mysql" switch value. As yet another example, if the configured switch value for the data to be migrated is "all", then the data migration statements associated with the "oracle" and "mysql" switch values ​​are selected, and the content comparison of the data to be migrated in the first target database and the second target database is performed asynchronously using the data migration statements associated with the "oracle" and "mysql" switch values.

[0079] In an exemplary embodiment, the process of obtaining the data to be migrated includes: calling all database tables in the MySQL database using a preset show table statement to obtain the corresponding table name information; iterating through all database tables using a preset show table statement to obtain initial table structure definition information, which includes field column names, data types of the field columns, default values, and additional information; and concatenating and converting the database tables and the initial table structure definition information to generate the data to be migrated. Specifically, the table name information of all database tables in the initial MySQL database is obtained. In this embodiment, a data table consists of three parts: table name, fields in the table, and records in the table. Designing the data table structure involves defining the data table name, determining which fields the data table contains, and the field names, field types, and widths of each field. This solution obtains the table name information of all database tables in the initial MySQL database, and can further obtain the initial table structure definition information of the corresponding data table based on the table name. The show table statement calls all database tables in the initial MySQL database to obtain the corresponding table name information. The show table statement can display all data tables in the current initial MySQL database, and the corresponding table name information can be directly obtained based on the displayed data tables. Then, the `desc + [table name]` statement is used to iterate through the corresponding database tables to obtain the initial table structure definition information. The `desc + [table name]` statement displays which fields are defined in the data table, the type and size of each field, which are primary keys, which have constraints, and whether each field has a default value, etc. This is equivalent to writing the `show table` and `desc + [table name]` statements as a Python script and importing it into the terminal. The script automatically connects to the MySQL database and executes database definition statements like `show tables` and `desc + [table name]` to obtain all table name information and table structure definition information. The database tables and the initial table structure definition information are then concatenated and transformed to generate the data to be migrated. In this embodiment, MySQL and Oracle, as two mainstream relational database systems, differ slightly in their usage. Although they are compatible with most SQL query statements, there are still some areas where they are not fully compatible. Since MySQL and Oracle are not fully compatible, the obtained database table and initial table structure definition information needs to be concatenated and transformed using corresponding conversion rules. This ensures that the data to be migrated conforms to the target table structure definition information required by the Oracle or MySQL database, thereby guaranteeing the integrity of the migrated database and improving data security.

[0080] In an exemplary embodiment, after migrating the data to be migrated from a first target database to a second target database using the selected data migration statement, the system further includes: acquiring the data migrated to the second target database, denoted as target migration data; extracting keywords or key fields from the target migration data, and sharding the target migration data based on the keywords or key fields. In this embodiment, the keywords or key fields in the target migration data include shard refresh parameters, which include: shard size, page size, and total data volume. Specifically, firstly, the start index and end index corresponding to the first page number in the database are determined according to the first page number and the page size. Then, the start index of the first page number in each database shard is determined according to the start index, page size, and total data volume. Finally, the end index of the first page number in each database shard is determined according to the end index, page size, and total data volume. Then, according to the start index and end index in each database shard, the corresponding data values ​​are obtained from each sorted database shard, and the data values ​​are combined to form a first intermediate data set. Specifically, in this embodiment, the first page number is defined as `pageIndex`, the database shard size is `sliceSize`, the page size is `pageSize`, and the total data volume is the total number of data stored in the database, defined as `total`. From the database configuration in the above scheme, the first page number `pageIndex` is 3, the database shard size `sliceSize` is 25, the page size `pageSize` is 4, and the total data volume `total` is 100. First, based on the first page number `pageIndex` and the page size `pageSize`, the starting index `startIndex` and the ending index `endIndex` corresponding to the first page number `pageIndex` in all data in the database are calculated, where: `startIndex = pageSize * (pageIndex - 1)`; `endIndex = pageSize * pageIndex - 1`. Since the first page number pageIndex is 3 and pageSize is 4, therefore: startIndex = 4 * (3 - 1) = 8; endIndex = 4 * 3 - 1 = 11; then the starting index of the third page corresponding to the first page number in the database is 8 and the ending index of the page is 11 in the sorted database.Then, based on the start index startIndex, end index endIndex, slice size sliceSize, and total data volume corresponding to the first page number, calculate the start index startIndexInSlice and end index endIndexInSlice of the first page number in each database slice, where: startIndexInSlice = sliceSize * startIndex / total; endIndexInSlice = sliceSize * endIndex / total; and startIndexInSlice is the value after rounding down, and endIndexInSlice is the value after rounding up. Since sliceSize is 25, total is 100, startIndex is 8, and endIndex is 11, therefore: startIndexInSlice = 25 * 8 / 100 = 2; endIndexInSlice = 25 * 11 / 100 = 2.75; after rounding endIndexInSlice up, endIndexInSlice is 3. Finally, according to the start and end indices in the database sharding, the corresponding data values ​​are obtained from each database shard, and the data values ​​form the first intermediate data set, completing the sharding of the target migration data.

[0081] It should be noted that the data migration system provided in the above embodiments and the data migration method provided in the above embodiments belong to the same concept. The specific way in which each module performs its operation has been described in detail in the method embodiments, and will not be repeated here. In practical applications, the data migration system provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above. This is not a limitation here.

[0082] In summary, this application provides a data migration system. First, it acquires the data to be migrated. Then, based on the business logic type of the data to be migrated, it performs business logic layer matching to determine the business logic layer of the data to be migrated. Next, it adds a proxy persistence layer between the business logic layer and a preset persistence layer, and uses business switches in the proxy persistence layer to configure switch values ​​for the data to be migrated. Finally, it selects a data migration statement based on the configured switch values ​​and uses the selected data migration statement to migrate the data to be migrated from a first target database to a second target database, thus enabling a persistent switching between the first and second target databases. Therefore, this system ensures code-free modification by embedding a proxy persistence layer. It achieves zero modification to the original logic code by adding a proxy layer between the original business logic layer and the persistence layer, forming a proxy persistence layer. This not only realizes logical separation at the code level but is also applicable to most business systems for code-free modification, ensuring the security and reliability of the data migration process. This system, by embedding a proxy persistence layer, not only achieves zero modification to existing code, ensuring security and reliability, but also reduces data migration costs. Furthermore, it supports switching between on / off data persistence methods, enabling online switching and dual-write asynchronous verification. In addition, the system's modifications throughout the data migration process are lightweight and imperceptible to users, making it suitable for most business systems requiring no code modification. Essentially, this system, based on the idea of ​​zero modification to existing logic code, achieves logical separation at the code level by adding a proxy persistence layer. Because it does not modify existing code, it ensures security and reduces data migration costs, making it suitable for most business systems requiring no code modification. If the first target database is a MySQL database and the second target database is an Oracle database, this system can migrate data from the MySQL database to the Oracle database. Conversely, if the first target database is an Oracle database and the second target database is a MySQL database, this system can migrate data from the Oracle database to the MySQL database.

[0083] Embodiments of this application also provide a computer device, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the electronic device to implement the data migration methods provided in the above embodiments. Figure 5 A schematic diagram of the structure of a computer device 500 is shown. (See also...) Figure 5 As shown, the computer device 500 includes: a processor 510, a memory 520, a power supply 530, a display unit 540, and an input unit 560.

[0084] The processor 510 is the control center of the computer device 500. It connects various components via interfaces and lines, and executes various functions of the computer device 500 by running or executing software programs and / or data stored in the memory 520, thereby providing overall monitoring of the computer device 500. In this embodiment, when the processor 510 calls the computer program stored in the memory 520, it executes... Figure 1 The method described herein. Optionally, processor 510 may include one or more processing units; preferably, processor 510 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. In some embodiments, the processor and memory may be implemented on a single chip; in some embodiments, they may also be implemented separately on independent chips.

[0085] The memory 520 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, various applications, etc.; the data storage area may store data created based on the use of the computer device 500, etc. In addition, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device, etc.

[0086] The computer device 500 also includes a power supply 530 (such as a battery) that supplies power to various components. The power supply can be logically connected to the processor 510 through a power management system, thereby enabling the management of charging, discharging, and power consumption.

[0087] The display unit 540 can be used to display information input by the user or information provided to the user, as well as various menus of the computer device 500. In this embodiment, it is mainly used to display the display interfaces of various applications in the computer device 500, and the text, pictures, and other objects displayed on the display interfaces. The display unit 540 may include a display panel 550. The display panel 550 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0088] Input unit 560 can be used to receive information such as numbers or characters input by the user. Input unit 560 may include touch panel 570 and other input devices 580. Touch panel 570, also known as touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 570).

[0089] Specifically, the touch panel 570 can detect user touch operations and the signals generated by these operations, converting them into touch point coordinates and sending them to the processor 510. It can also receive and execute commands from the processor 510. Furthermore, the touch panel 570 can be implemented using various types of sensors, including resistive, capacitive, infrared, and surface acoustic wave sensors. Other input devices 580 can include, but are not limited to, one or more of the following: a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick.

[0090] Of course, the touch panel 570 can cover the display panel 550. When the touch panel 570 detects a touch operation on or near it, it transmits the information to the processor 510 to determine the type of touch event. Subsequently, the processor 510 provides corresponding visual output on the display panel 550 according to the type of touch event. Although in Figure 5 In this embodiment, the touch panel 570 and the display panel 550 are two separate components to realize the input and output functions of the computer device 500. However, in some embodiments, the touch panel 570 and the display panel 550 can be integrated to realize the input and output functions of the computer device 500.

[0091] The computer device 500 may also include one or more sensors, such as pressure sensors, gravity acceleration sensors, proximity sensors, etc. Of course, depending on the specific application requirements, the computer device 500 may also include other components such as a camera.

[0092] This application also provides a computer-readable storage medium storing instructions that, when executed by one or more processors, enable the device to perform the functions described in this application. Figure 1 The method described.

[0093] It will be understood by those skilled in the art that Figure 5 This is merely an example of a computer device and does not constitute a limitation on the device. The device may include more or fewer components than illustrated, or a combination of certain components, or different components. For ease of description, the above sections are divided into modules (or units) according to their functions and described separately. Of course, in implementing this application, the functions of each module (or unit) can be implemented in one or more software or hardware components.

[0094] Those skilled in the art will understand that this application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application, and should be understood to be achievable by computer program instructions for each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams. These computer program instructions may be applied to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] It should be understood that although terms such as "first," "second," etc., may be used to describe target databases in the embodiments of this application, these presuppositions should not be limited to these terms. These terms are only used to distinguish the presuppositions from each other. For example, without departing from the scope of the embodiments of this application, the first target database may also be referred to as the second target database, and similarly, the second target database may also be referred to as the first target database.

[0096] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A data migration method, characterized in that, The method includes the following steps: The process involves acquiring the data to be migrated and performing business logic layer matching based on the business logic type of the data to be migrated to determine the business logic layer of the data to be migrated. The acquisition of the data includes: using the `show table` statement to call all database tables in the initial MySQL database and obtaining the corresponding table name information; displaying all data tables in the current initial MySQL database using the `show table` statement and directly obtaining the corresponding table name information based on the displayed data tables; then looping through the corresponding database tables using the `desc+[table name]` statement to obtain the initial table structure definition information; concatenating and converting the database table and the initial table structure definition information to generate the data to be migrated; and displaying the specific fields defined in the data table using the `desc+[table name]` statement, as well as the type, size, primary key, and constraints of each field, and whether each field has a default value. A proxy persistence layer is added between the business logic layer and the preset persistence layer, and the business switches in the proxy persistence layer are used to configure the switch values ​​of the data to be migrated. Based on the configured switch value, a data migration statement is selected, and the selected data migration statement is used to migrate the data to be migrated from the first target database to the second target database, so that the data to be migrated can be persistently switched between the first target database and the second target database. The process of adding a proxy persistence layer between the business logic layer and the preset persistence layer includes: The logic interface is created by creating a pre-defined object to simulate the generation of access objects in the target container framework, and the simulated access objects are then transmitted to the pre-defined container. The access objects in the preset container are dynamically managed so that the access objects in the preset container are logically enhanced after dynamic management; Obtain parameters for different target types, and instantiate the dynamically managed access objects according to the preset construction method. Based on the instantiation result, add a proxy persistence layer between the business logic layer and the preset persistence layer.

2. The data migration method according to claim 1, characterized in that, The process of configuring the switch value of the data to be migrated using the business switch in the proxy persistence layer includes: Add a business switch to the proxy persistence layer, and use the business switch to configure the switch value of the data to be migrated, selecting the switch value corresponding to the data to be migrated; The switch values ​​used for selection include at least one of the following: oracle, mysql, all.

3. The data migration method according to claim 2, characterized in that, The process of selecting a data migration statement based on the configured switch value and using the selected data migration statement to migrate the data to be migrated from the first target database to the second target database includes: Obtain the configured switch value for the number to be migrated, and record it as the target switch value; Select a data migration statement associated with the target switch value, and use the selected data migration statement to migrate the data to be migrated from the first target database to the second target database; or, Select a data migration statement associated with the target switch value, and asynchronously perform a comparison of the data to be migrated in the first target database and the second target database using the selected data migration statement.

4. The data migration method according to claim 1 or 3, characterized in that, The first target database is an Oracle database and the second target database is a MySQL database; or, the first target database is a MySQL database and the second target database is an Oracle database.

5. The data migration method according to claim 1, characterized in that, After migrating the data to be migrated from the first target database to the second target database using the selected data migration statement, the method further includes: Obtain the data that has been migrated to the second target database, and denote it as the target migration data; Extract keywords or key fields from the target migration data, and segment the target migration data based on the keywords or key fields.

6. A data migration system, characterized in that, The system includes: The data acquisition module is used to acquire the data to be migrated, including: calling all database tables in the initial MySQL database using the `show table` statement to obtain the corresponding table name information; specifically, displaying all data tables under the current initial MySQL database using the `show table` statement and directly obtaining the corresponding table name information based on the displayed data tables; then looping through the corresponding database tables using the `desc+[table name]` statement to obtain the initial table structure definition information; concatenating and converting the database tables and the initial table structure definition information to generate the data to be migrated; specifically, displaying the specific fields defined in the data table using the `desc+[table name]` statement, as well as the type, size, primary key, and constraints of each field, and whether each field has a defined default value; The data matching module is used to perform business logic layer matching based on the business logic type of the data to be migrated, and to determine the business logic layer of the data to be migrated. A switch value configuration module is used to add a proxy persistence layer between the business logic layer and the preset persistence layer, and to configure the switch values ​​of the data to be migrated using the business switches in the proxy persistence layer. The process of adding a proxy persistence layer between the business logic layer and the preset persistence layer includes: simulating the generation of access objects in the target container framework through a preset object creation logic interface, and transmitting the simulated access objects to the preset container; dynamically managing the access objects in the preset container so that the access objects in the preset container are logically enhanced after dynamic management; obtaining parameters of different target types, and instantiating the dynamically managed access objects according to a preset construction method, and adding a proxy persistence layer between the business logic layer and the preset persistence layer based on the instantiation result. The data migration module is used to select a data migration statement according to the configured switch value, and use the selected data migration statement to migrate the data to be migrated from the first target database to the second target database, so that the data to be migrated can be persistently switched between the first target database and the second target database.

7. A computer device, characterized in that, The computer device includes: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the computer device to implement the data migration method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by the computer's processor, causes the computer to perform the data migration method as described in any one of claims 1 to 5.

Images