Data migration method and device, computer program product, medium and processor

Abstract

The invention discloses a data migration method and device, a computer program product, a medium and a processor, and relates to the technical field of data migration. The method comprises the steps that original system information, target system information and data analysis information of to-be-migrated data are obtained, the original system information is used for representing a data architecture of an original system, and the target system information is used for representing a data architecture of a target system; the data analysis information is information generated after to-be-migrated data in an original system is analyzed by an analysis tool; determining a data migration strategy of the to-be-migrated data based on the original system information, the target system information and the data analysis information; and migrating the to-be-migrated data in the original system to the target system according to the data migration strategy. By means of the technical scheme, automatic data migration is achieved, migration cost is greatly reduced, powerful support is provided for achieving unification and standardization of database technology stacks, and faster and more stable development of enterprises under a new strategy is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and in particular to a data migration method, device, computer program product, medium and processor. Background Art

[0002] As the strategy of Xinchuang (innovation in information technology applications) deepens, domestic enterprises are facing an important historical node of independent control of basic software and hardware, which is not only a test of the technical strength of enterprises, but also the key to achieving long-term stable development and ensuring data security. This trend is particularly evident in the database field. With the continuous enrichment of business scenarios and load, enterprises have an increasingly strong demand for multi-mode NoSQL databases.

[0003] MongoDB, with its unique document storage model and powerful query capabilities, stands out among NoSQL databases and has become the first choice for many companies. However, with the rapid expansion of enterprise scale and in-depth development of business, the coexistence of multiple database technology stacks has gradually exposed the complexity of operation and maintenance and management challenges. This not only increases the IT costs of enterprises, but may also affect the stable operation of the business.

[0004] In this context, OceanBase, as an autonomous and controllable distributed relational database, has been favored by more and more large enterprises due to its high performance, high availability, and scalability. Unifying the database technology stack has become an important way for enterprises to simplify operation and maintenance management, reduce costs, and improve overall data management efficiency. Therefore, migrating MongoDB to OceanBase has become an urgent need for many enterprises.

[0005] However, migrating MongoDB data to OceanBase is not an easy task. This process involves multiple steps, such as data structure conversion, data type compatibility processing, and data export and import. Since OceanBase and MongoDB have significant differences in data storage and query mechanisms, a series of technical challenges need to be solved during the migration process. Currently, there is no direct migration tool between OceanBase and MongoDB, which further increases the complexity and risk of migration.

[0006] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0007] The embodiments of the present invention provide a data migration method, device, computer program product, medium and processor to at least solve the technical problems of high complexity, high risk and high technical requirements in the migration of MongoDB data to OceanBase database in the related art.

[0008] According to one aspect of an embodiment of the present invention, a data migration method is provided, comprising the following steps: obtaining original system information, target system information, and data analysis information of the data to be migrated, the original system information being used to represent the data architecture of the original system, the target system information being used to represent the data architecture of the target system, and the data analysis information being information generated after an analysis tool analyzes the data to be migrated in the original system; determining a data migration strategy for the data to be migrated based on the original system information, the target system information, and the data analysis information; and migrating the data to be migrated in the original system to the target system according to the data migration strategy.

[0009] Optionally, the method also includes the following steps: obtaining code analysis information, the code analysis information being information generated by an analysis tool after analyzing the application code corresponding to the original system; generating a code adjustment strategy based on the code analysis information; and adjusting the application code corresponding to the original system to the application code corresponding to the target system based on the code adjustment strategy.

[0010] Optionally, the original system includes a MongoDB system, and based on the original system information, the target system information and the data analysis information, a data migration strategy for the data to be migrated is determined, including the following steps: establishing data conversion rules based on the original system information, the target system information, the database information in the data analysis information, the collection information, the structure information of the MongoDB document, and the index information; generating a migration analysis result based on the data conversion rule, the migration analysis result being used to represent the migration assessment result and conversion suggestion of the MongoDB document; and determining the data migration strategy for the data to be migrated based on the migration analysis result.

[0011] Optionally, based on the migration analysis results, determining a data migration strategy for the data to be migrated includes the following steps: based on the migration analysis results, determining a document selection strategy in the data migration strategy; wherein the document selection strategy is used to determine whether to migrate MongoDB documents to the target system.

[0012] Optionally, based on the migration analysis result, determining the data migration strategy includes the following steps: based on the migration analysis result, determining the storage method selection strategy in the data migration strategy; wherein the storage method selection strategy is used to determine the storage method of the MongoDB document after migration.

[0013] Optionally, the original system includes a MongoDB system, and generating a code adjustment strategy based on code analysis information includes the following steps: generating a code adjustment strategy based on interaction information, call information, and associated document information in the code analysis information; wherein the interaction information is used to represent a code segment in the application code that interacts with the MongoDB system, the call information is used to represent a code segment in the application code that calls the MongoDB system, and the associated document information is used to represent MongoDB document information in the MongoDB system that is accessed by the application code.

[0014] According to another aspect of an embodiment of the present invention, a data migration device is also provided, including: an acquisition module, the acquisition module is used to acquire original system information, target system information, and data analysis information of the data to be migrated, the original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is information generated after the analysis tool analyzes the data to be migrated in the original system; a determination module, the determination module is used to determine a data migration strategy for the data to be migrated based on the original system information, the target system information and the data analysis information; and a migration module, the migration module is used to migrate the data to be migrated in the original system to the target system according to the data migration strategy.

[0015] According to another aspect of the present application, a computer program product is also provided, comprising computer instructions, which implement the steps of the above-mentioned data migration method when executed by a processor.

[0016] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned data migration method when running.

[0017] According to yet another aspect of an embodiment of the present invention, a processor is provided, and the processor is used to run a program, wherein the program is configured to execute any of the aforementioned data migration methods when running.

[0018] In the embodiment of the present invention, the original system information, target system information, and data analysis information of the data to be migrated are obtained, the original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is the information generated by the analysis tool after analyzing the data to be migrated in the original system; based on the original system information, the target system information, and the data analysis information, the data migration strategy of the data to be migrated is determined; according to the data migration strategy, the data to be migrated in the original system is migrated to the target system. By analyzing the existing application code and document structure, a detailed data migration strategy can be provided in combination with the data volume, and at the same time, efficient data analysis tools and migration tools are used to realize automatic data migration, which greatly reduces the migration cost, provides strong support for realizing the sameness and standardization of the database technology stack, and realizes faster and more stable development of enterprises under the new start-up strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 is a hardware structure block diagram of a computer terminal according to a data migration method according to one embodiment of the present invention;

[0021] Figure 2 is a flow chart of a data migration method according to one optional embodiment of the present invention;

[0022] Figure 3 is a structural block diagram of a data migration device according to one embodiment of the present invention;

[0023] Figure 4 It is a flowchart of a data migration method according to one optional embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0027] NoSQL database: NoSQL database (Not Only Structured Query Language, non-relational database) is designed to process large-scale data, especially data with low structure or data model that may change frequently.

[0028] MongoDB: MongoDB is a popular document-oriented NoSQL database that is widely used in applications that need to handle semi-structured data and dynamic schemas, such as social media, content management, and the Internet of Things.

[0029] OceanBase: OceanBase is a distributed relational database system designed to support ultra-large-scale data storage and complex queries.

[0030] ASM (JAVA bytecode manipulation and analysis tool): Another Small Machine, ASM is a tool library for analyzing, manipulating and generating JAVA bytecodes. It provides a set of efficient APIs (Application Programming Interface) that allow developers to programmatically modify the bytecodes of JAVA classes, modify class structures, insert methods, optimize bytecodes, and perform other operations.

[0031] DataX: Data Exchange is an efficient data migration tool that provides a flexible plug-in architecture and supports data exchange between various data sources.

[0032] ORM (Object Relational Mapping): ORM is a programming technology that maps objects in object-oriented programming to tables in a relational database.

[0033] According to one embodiment of the present invention, an embodiment of a data migration method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] The method embodiment can be executed in a computer terminal or a similar computing device. For example, if the method is executed in a computer terminal, Figure 1 As shown, the computer terminal may include one or more processors 102 (the processor may include but is not limited to a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP) chip, a microprocessor (MCU), a field programmable logic device (FPGA), a neural network processor (NPU), a tensor processor (TPU), an artificial intelligence (AI) type processor, etc.) and a memory 104 for storing data. Optionally, the above-mentioned computer terminal may also include a transmission device 106 for communication functions, an input and output device 108, and a display 110. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal may also include more or fewer components than the above-mentioned structure description, or have a configuration different from the above-mentioned structure description.

[0035] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the data migration method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, the above-mentioned data migration method is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0036] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0037] The display 110 may be, for example, a touch screen liquid crystal display (LCD). The liquid crystal display may enable a user to interact with a user interface of the mobile terminal. In some embodiments, the mobile terminal has a graphical user interface (GUI), and a user may interact with the GUI by finger contact and / or gestures on a touch-sensitive surface, wherein the human-computer interaction functions here may optionally include the following interactions: creating web pages, drawing, word processing, making electronic documents, games, video conferencing, instant messaging, sending and receiving emails, call interfaces, playing digital videos, playing digital music, and / or web browsing, etc. The executable instructions for executing the above human-computer interaction functions are configured / stored in a computer program product or a readable storage medium executable by one or more processors.

[0038] This embodiment provides a data migration method running on the above-mentioned computer terminal. Figure 2 is a flow chart of a data migration method according to one embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0039] Step S21, obtaining original system information, target system information, and data analysis information of the data to be migrated, wherein the original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is information generated after the analysis tool analyzes the data to be migrated in the original system;

[0040] Specifically, the data to be migrated is the data in the original system database that needs to be migrated to the target system database. The original system database and the target system database are databases with different data models and types. The original system information includes data architecture (collections, document structures, indexes, associations, data distribution), storage details (data volume, data type, document size distribution) and usage patterns, etc. The target system information may perform the above information statistics and compare it with the original system information. The differences between the two systems can be identified through analysis tools to generate data analysis information.

[0041] In step S21, the database and collection information are queried through the management command or API of the original system, and the original system information is obtained by using ORM or direct SQL query; the database structure of the target system is queried through the management interface or API of the target system; the original data is deeply analyzed using analysis tools to generate relevant reports. Based on the above information and related reports, developers can formulate more reasonable and efficient data migration strategies to ensure the success of data migration.

[0042] Step S22, determining a data migration strategy for the data to be migrated based on the original system information, the target system information and the data analysis information;

[0043] Specifically, the acquired original system information, target system information and data analysis information are processed to determine the data migration strategy, which includes comparing data models, formulating data conversion rules, analyzing data volume and performance requirements, formulating migration plans, applying code transformation and other specific strategies.

[0044] Step S23: Migrate the data to be migrated in the original system to the target system according to the data migration strategy.

[0045] Specifically, the data migration strategy includes multiple steps, and the migration of the data to be migrated in the original system to the target system needs to be carried out in sequence according to the logical steps in the data migration strategy.

[0046] In step S23, the data migration implementation process needs to perform data preprocessing in sequence. Preprocess the original data according to the data migration strategy (such as data cleaning, data type conversion, cleaning or standardizing non-standard characters, etc.); use the data export tool (which can be the export tool provided by the database, or the custom export script edited by the developer) to export the data into an intermediate storage format (such as a JSON file or a CSV file); formulate data conversion rules to convert the exported data into the corresponding data format in the target system (such as splitting the document, converting the JSON field into a specific column, adjusting the data type and its compatibility, etc.); use the import tool supported by the target system or the custom script edited by the developer to import the data into the target system; generate application code transformation suggestions, adjust the data access logic in the application (such as updating the PRM configuration, modifying the SQL query statement, and adjusting the database interaction code); test, monitor and adjust. Through the above logical steps, ensure that the data is migrated stably, efficiently and accurately.

[0047] Through the above steps, the original system information, target system information, and data analysis information of the data to be migrated are obtained. The original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is the information generated by the analysis tool after analyzing the data to be migrated in the original system; based on the original system information, target system information, and data analysis information, the data migration strategy of the data to be migrated is determined; according to the data migration strategy, the data to be migrated in the original system is migrated to the target system. By analyzing the existing application code and document structure, a detailed data migration strategy can be provided in combination with the data volume. At the same time, efficient data analysis tools and migration tools are used to realize automatic data migration, which greatly reduces the migration cost, provides strong support for the realization of the same and standardized database technology stack, and realizes faster and more stable development of enterprises under the new start-up strategy.

[0048] Optionally, the method further comprises the following steps:

[0049] Step S24, obtaining code analysis information, where the code analysis information is information generated after the analysis tool analyzes the application code corresponding to the original system;

[0050] Specifically, the code analysis information obtained by developers after analyzing the application corresponding to the original system through analysis tools can further help developers understand the principles and methods of interaction between the application and the original system database.

[0051] In this embodiment, an analysis tool is used to scan the source code of the application, identify data access points (query, write, update, delete and other operation points), and determine the application's access mode to data, including query frequency, data read and write ratio, transaction processing method and data operation performance requirements; identify the dependencies in the code on the data model, query syntax, data type and storage characteristics in the original system; and generate a code analysis report, that is, generate code analysis information.

[0052] Step S25, generating a code adjustment strategy based on the code analysis information;

[0053] Specifically, the code adjustment strategy is a strategy for adjusting the code in the data migration strategy. During the data migration process, the application is adjusted and migrated according to the generated code adjustment strategy.

[0054] In step S25, based on the code analysis information, the use of object-relational mapping, query syntax conversion, data model adjustment, stored procedures and functions adjustment, and code segments that directly use the original system driver for operation are identified and adjusted. Through the above specific operations, the application can be guaranteed to run normally after data migration while retaining or improving its performance.

[0055] Step S26: Based on the code adjustment strategy, the application code corresponding to the original system is adjusted to the application code corresponding to the target system.

[0056] Specifically, if the syntax, data model, storage, function, document and other data structures of the original system and the target system are different, the application code format corresponding to the original system and the application code format corresponding to the target system will also be different. The application code should be adjusted so that the application can be migrated from the original system to the target system to ensure business continuity and data integrity.

[0057] In step S26, the adjustment of the application code of the original system needs to go through the following specific steps: update the original system data block link code; modify the data access code, convert the query statement of the original system into the specific API of the target system, adjust the operations such as insert, update, delete to ensure that they match the data model and operation mode of the target system, replace or convert the data type of the original system into the data type supported by the target system; reconstruct the document model of the original system into the table model of the target system; organize the code and replace all direct calls to the original system with calls to the target system. Based on the code adjustment strategy, the application of the original system is smoothly migrated to the target system.

[0058] Optionally, in step S21-step S22, the original system includes a MongoDB system, and determining a data migration strategy for the data to be migrated based on the original system information, the target system information, and the data analysis information includes the following steps:

[0059] Step S221, establishing data conversion rules based on the original system information, the target system information, the database information, the collection information, the structure information of the MongoDB document, and the index information in the data analysis information;

[0060] Specifically, the data model of the MongoDB system includes database information, collection information, document structure information, index information, etc. The collection information needs to list all the collections of the MongoDB system, analyze the document structure of each collection, identify the fields and data types, and understand the indexes on each collection (including fonts, types, and uses, etc.); after analyzing and understanding the MongoDB system, it is also necessary to analyze and understand the above information and corresponding characteristics of the target system, and formulate and establish data conversion rules in combination with the data analysis information.

[0061] In step S221, establishing data conversion rules includes establishing mapping rules from MongoDB system fields to target system table columns, converting the data types of the MongoDB system to the data types of the target system, deciding how to handle nested documents and array fields in the MongoDB system, formulating corresponding index strategies in the target system based on the index information of the MongoDB system, and considering the performance and compatibility of the MongoDB system and the target system.

[0062] It should be noted that when establishing data conversion rules, you also need to pay attention to the conversion of MongoDB system date types, the processing of array fields, the processing of null values ​​and default values, and unique constraints. For example, the date type of the MongoDB system may need to be converted to the DATE or TIMESTAMP type in the target system; if there are a large number of array fields in the MongoDB system collection, consider using other fields to store them in the target system or creating an independent table for storage; convert the unique index method in the MongoDB system to the unique constraint or unique index method in the target system.

[0063] Step S222: Generate a migration analysis result based on the data conversion rule, where the migration analysis result is used to represent the migration assessment result and conversion suggestion of the MongoDB document;

[0064] Specifically, the migration analysis results are used to evaluate how MongoDB documents adapt to the target system documents. After the system processes the relevant data of the MongoDB system according to the data conversion rules, it generates the migration analysis results based on the processing process. The migration analysis results include the mapping relationship between the MongoDB system and the target system fields, a data type conversion list, an explanation of index conversion, a processing strategy for complex data structures, an assessment of the impact of data migration on query performance, a list of code snippets that need to be adjusted and modification suggestions, a data migration risk report, and a test plan.

[0065] In step S222, the contents of the above migration analysis results are summarized to provide a comprehensive migration assessment report. After analyzing the above contents, the migration assessment report evaluates the feasibility, expected effects and potential risks of data migration and obtains an assessment result. At the same time, for the data migration issues involved in the migration assessment report, detailed conversion suggestions are provided for each collection, including field mapping, data type conversion, index reconstruction, and code adjustment strategy suggestions.

[0066] Step S223: determining a data migration strategy for the data to be migrated based on the migration analysis result.

[0067] Specifically, the migration analysis results can provide detailed data migration suggestions, evaluate the data volume and complexity based on the migration analysis results, ensure the consistency of data before and after migration, determine the time window for data migration, determine the data migration tool and select the tool configuration. The migration analysis results can provide the above-mentioned suggestions for the problems, and the developers can determine the data migration strategy for the data to be migrated based on the problems listed in the migration analysis results.

[0068] In one embodiment of the present application, the data migration strategy is set to: phased migration, first migrate a part of the data to the target system, perform preliminary testing and adjustments, and then gradually migrate the remaining data; incremental migration, during the migration process, continuously synchronize newly generated data to ensure the real-time and consistency of the data; pre-migration environment setting, set up a pre-migration environment outside the production environment to test the effectiveness of code adjustments and data migration strategies; compare data before and after migration to verify data integrity and optimize data storage structure and query performance according to the characteristics of the target system.

[0069] It should be noted that the data migration strategy can be decided by the developers independently. The migration method, migration amount, migration time, and testing and monitoring after data migration are all decided by the developers to ensure the efficiency, accuracy and security of data migration, thereby providing support for the smooth transition of the business.

[0070] Optionally, in step S223, based on the migration analysis result, determining a data migration strategy for the data to be migrated includes the following steps:

[0071] Step S2231, determining a document selection strategy in a data migration strategy based on the migration analysis result;

[0072] Specifically, document selection refers to how to select, process, and organize documents to adapt to the relational database structure of the target system during the data migration process from the MongoDB system to the target system. The document selection strategy needs to be formulated based on the migration analysis results.

[0073] Among them, the document selection strategy is used to determine whether to migrate MongoDB documents to the target system. According to the migration analysis results, the complexity and usage frequency of each document in the MongoDB system are evaluated, and the documents are classified. For complex documents, how to split nested documents or array fields into multiple tables in the target system is determined; at the same time, the data volume of the document is evaluated, and the performance optimization strategy of the target system is considered to finally decide whether to migrate MongoDB documents to the target system.

[0074] Optionally, in step S223, based on the migration analysis result, determining a data migration strategy includes the following steps:

[0075] Step S2232, determining a storage mode selection strategy in the data migration strategy based on the migration analysis result;

[0076] Specifically, the database of the target system can provide multiple storage options to adapt to different data characteristics and query requirements. According to the differences in data storage methods between the MongoDB system and the target system provided in the migration analysis results, select the appropriate storage method for data migration.

[0077] Among them, the storage method selection strategy is used to determine the storage method after the MongoDB document migration. The storage method selection strategy needs to consider the data characteristics and performance of the MongoDB system and the target system, including data scale, data structure, and data access mode; based on data access frequency and query complexity, select appropriate storage methods and index strategies to maintain data consistency and reliability and meet data redundancy storage and data backup storage requirements; at the same time, considering the convenience of data maintenance, select a storage method that is easy to maintain data to reduce complex data queries and data management.

[0078] In one embodiment of the present application, based on the migration analysis results, the available storage methods include single table storage, partition table storage, and associated table storage, which differ in the simplicity and complexity of the data structure and data volume. By determining the storage method selection strategy, it can be ensured that the storage of data in the target system meets the requirements and can give full play to the performance advantages of the target system.

[0079] Optionally, in step S25, the original system includes a MongoDB system, and based on the code analysis information, a code adjustment strategy is generated, including the following steps:

[0080] Step S251, generating a code adjustment strategy based on the interaction information, call information, and associated document information in the code analysis information;

[0081] Specifically, the process of generating the code adjustment strategy requires analyzing the interaction between the application and the MongoDB system, the code that calls the MongoDB system API or method, and the code that accesses the MongoDB system document. By evaluating the above information and comparing it with the corresponding information in the target system, the connection between the MongoDB system and the target system application is achieved in terms of the interaction between the application and the system, the system calls, and the associated document data model. A code transformation plan is formulated and a code adjustment strategy is generated.

[0082] Among them, interaction information is used to represent the code segments in the application code that interact with the MongoDB system, call information is used to represent the code segments in the application code that call the MongoDB system, and associated document information is used to represent the MongoDB document information accessed by the application code in the MongoDB system. The code adjustment strategy specifically includes modifying, querying, adjusting, data processing and logic changes to the relevant codes in the above information, which can ensure that the application can still run efficiently after the database is upgraded to meet business needs, while reducing risks and uncertainties in the transformation process.

[0083] Combination Figure 4 As shown, the present application also provides a preferred embodiment of a data migration method, which is used to migrate data in a MongoDB system to an OceanBase database system. The analysis and migration tool uses DataX, the development language is JAVA or Python, and the operating system is a Linux system, a Windows system, or a Mac system. Figure 4 The flowchart for migrating data from the MongoDB system to the OceanBase database system includes the following steps:

[0084] Step 1: Develop analysis tools, use the official driver of the MongoDB system or the ORM (object-relational mapping) library that supports the MongoDB system to connect to the target database; query the list of databases in the MongoDB system and the list of collections in each database; for each collection, obtain the structure information of its documents; query the index information of each collection; analyze the application code, identify the specific scenarios and functions of using the MongoDB system, and collect relevant document information; formulate specific conversion rules for conversion to the OceanBase database system based on the documents, indexes and application code usage scenarios of the MongoDB system; develop an output module to convert the analysis results into a migration plan recommendation document, and output it in .csv file format;

[0085] Among them, preferably, JAVA development analysis tools are used to introduce the official dependency SDK package of the MongoDB system; the bytecode analysis framework (ASM) is used to read the java application code (different analysis tools are selected according to different development languages) to analyze the Mongodb scenarios specifically used by the application code; according to MongoDB's documents, indexes, data size and application code usage scenarios, specific conversion rules for conversion to OceanBase are formulated, and the converted results are output to a CSV file.

[0086] Step 2: Use the developed analysis tools to conduct in-depth analysis of the MongoDB system database and application code to be converted; based on the analysis results, generate a detailed migration solution recommendation document;

[0087] Step 3: Evaluate each MongoDB document to determine whether to migrate it to a single table or multiple tables in the OceanBase database system; decide whether to use JSON fields for storage or fields with multiple data types; after review and discussion, develop a final document migration plan;

[0088] Step 4: Use the migration tool DataX to migrate data. If it involves migrating a single MongoDB document to multiple OceanBase tables, you need to develop an Output plug-in that adapts to the OceanBase multi-table structure.

[0089] Step 5: Adjust the application code according to the migration plan to ensure compatibility with OceanBase's data model and query language;

[0090] Step 6: Conduct data integrity, performance testing, and application functionality testing;

[0091] Step 7: Put the migrated system into production environment, monitor and provide feedback, and deal with possible problems and optimization needs in a timely manner.

[0092] It can be seen from the above description that the data migration method in this embodiment has the following beneficial effects:

[0093] 1) Accurate migration planning: By analyzing the existing application code and the document structure in the original system, combined with the data volume, detailed application code transformation and structure conversion suggestions are provided. Accurate planning can ensure that all data and structure adjustments are fully considered during the migration process, reducing compatibility issues and performance bottlenecks after migration.

[0094] 2) Efficient use of migration tools: The efficient data migration tools are used to realize the automatic migration of the data to be migrated in the original system. The migration tools have the characteristics of high performance and strong reliability. They can handle large amounts of data during the migration process, reduce the need for manual intervention, improve migration efficiency, and reduce the migration cost and time of the enterprise.

[0095] 3) Reduce technical difficulties: Difficulties encountered during data migration, such as data consistency, data type conversion, and complex structure adjustment, can be solved through careful planning and tool support, ensuring that data remains consistent and complete during the migration process, reducing technical difficulty and risk.

[0096] 4) Support strategic goals: By achieving the unification and standardization of the database technology stack, it not only solves the migration problem at the technical level, but also promotes the company to achieve faster and more stable development under the new venture strategy, pays attention to improving the overall technical level and enhancing market competitiveness.

[0097] 5) Reduce migration costs: The use of automated data migration tools and detailed transformation suggestions significantly reduces the labor and time costs during the migration process.

[0098] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0099] In this embodiment, a data migration device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0100] Figure 3 is a structural block diagram of a data migration device according to one embodiment of the present invention. Figure 3 As shown, the device includes: an acquisition module, the acquisition module is used to acquire original system information, target system information, and data analysis information of the data to be migrated, the original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is information generated after the analysis tool analyzes the data to be migrated in the original system; a determination module, the determination module is used to determine the data migration strategy of the data to be migrated based on the original system information, the target system information and the data analysis information; a migration module, the migration module is used to migrate the data to be migrated in the original system to the target system according to the data migration strategy.

[0101] Through the above device, the original system information, target system information, and data analysis information of the data to be migrated are obtained. The original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is the information generated by the analysis tool after analyzing the data to be migrated in the original system; based on the original system information, the target system information, and the data analysis information, the data migration strategy of the data to be migrated is determined; according to the data migration strategy, the data to be migrated in the original system is migrated to the target system. By analyzing the existing application code and document structure, a detailed data migration strategy can be provided in combination with the data volume. At the same time, efficient data analysis tools and migration tools are used to realize automatic data migration, which greatly reduces the migration cost, provides strong support for the realization of the same and standardization of the database technology stack, and realizes faster and more stable development of enterprises under the new start-up strategy.

[0102] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0103] Optionally, the data migration device provided in the present application may also include other modules. For example, the data migration device may also include a cleaning module, which cleans and preprocesses the original data before data migration, including data format standardization, deduplication or invalid data, filling missing values, etc., to ensure the quality of migrated data; the data migration device may also include a conversion module, which is used to convert the data format and structure of the original system into the format and structure of the target system, including field mapping, data type conversion, and complex data structure processing.

[0104] The embodiment of the present application further provides a readable storage medium. Optionally, in this embodiment, the readable storage medium can be used to store the program code executed by the data migration method provided in the first embodiment.

[0105] Optionally, in this embodiment, the readable storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0106] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of the data migration method.

[0107] Optionally, in this embodiment, the computer program is configured to perform the following steps:

[0108] Step S1, obtaining original system information, target system information, and data analysis information of the data to be migrated, wherein the original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is information generated after the analysis tool analyzes the data to be migrated in the original system;

[0109] Step S2, determining a data migration strategy for the data to be migrated based on the original system information, the target system information and the data analysis information;

[0110] Step S3: Migrate the data to be migrated in the original system to the target system according to the data migration strategy.

[0111] The readable storage medium provided in the present application stores a computer program, wherein the computer program is configured to execute the steps of any of the above-mentioned data migration method embodiments when running.

[0112] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0113] Step S1, obtaining original system information, target system information, and data analysis information of the data to be migrated, wherein the original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is information generated after the analysis tool analyzes the data to be migrated in the original system;

[0114] Step S2, determining a data migration strategy for the data to be migrated based on the original system information, the target system information and the data analysis information;

[0115] Step S3: Migrate the data to be migrated in the original system to the target system according to the data migration strategy.

[0116] An embodiment of the present invention further provides a processor, which is configured to run a computer program to execute the steps in any of the above data migration method embodiments.

[0117] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0118] Step S1, obtaining original system information, target system information, and data analysis information of the data to be migrated, wherein the original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is information generated after the analysis tool analyzes the data to be migrated in the original system;

[0119] Step S2, determining a data migration strategy for the data to be migrated based on the original system information, the target system information and the data analysis information;

[0120] Step S3: Migrate the data to be migrated in the original system to the target system according to the data migration strategy.

[0121] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0122] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0123] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0125] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0126] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0128] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data migration method, characterized in that: The method comprises the following steps: Obtaining original system information, target system information, and data analysis information of the data to be migrated, wherein the original system information is used to represent the data architecture of the original system, the target system information is used to represent the data architecture of the target system, and the data analysis information is information generated by an analysis tool after analyzing the data to be migrated in the original system; Determining a data migration strategy for the data to be migrated based on the original system information, the target system information, and the data analysis information; According to the data migration strategy, the data to be migrated in the original system is migrated to the target system.

2. The method according to claim 1, characterized in that The method further comprises the following steps: Acquire code analysis information, where the code analysis information is information generated by the analysis tool after analyzing the application code corresponding to the original system; Based on the code analysis information, generate a code adjustment strategy; Based on the code adjustment policy, the application code corresponding to the original system is adjusted to the application code corresponding to the target system.

3. The method according to claim 1 or 2, characterized in that: The original system includes a MongoDB system, and determining a data migration strategy for the data to be migrated based on the original system information, the target system information, and the data analysis information includes the following steps: Establishing data conversion rules based on the original system information, the target system information, the database information, the collection information, the structure information of the MongoDB document, and the index information in the data analysis information; Based on the data conversion rule, a migration analysis result is generated, where the migration analysis result is used to represent a migration assessment result and a conversion suggestion for the MongoDB document; Based on the migration analysis result, a data migration strategy for the data to be migrated is determined.

4. The method according to claim 3, characterized in that Determining a data migration strategy for the data to be migrated based on the migration analysis result includes the following steps: Determining a document selection strategy in the data migration strategy based on the migration analysis result; The document selection strategy is used to determine whether to migrate the MongoDB document to the target system.

5. The method according to claim 3, characterized in that: Determining the data migration strategy based on the migration analysis result includes the following steps: Determining a storage mode selection strategy in the data migration strategy based on the migration analysis result; The storage mode selection strategy is used to determine the storage mode of the MongoDB document after migration.

6. The method according to claim 2, characterized in that The original system includes a MongoDB system, and based on the code analysis information, a code adjustment strategy is generated, including the following steps: Generate the code adjustment strategy based on the interaction information, call information, and associated document information in the code analysis information; Among them, the interaction information is used to represent the code segment in the application code that interacts with the MongoDB system, the call information is used to represent the code segment in the application code that calls the MongoDB system, and the associated document information is used to represent the MongoDB document information accessed by the application code in the MongoDB system.

7. A data migration device, characterized in that: include: an acquisition module, the acquisition module being used to acquire original system information, target system information, and data analysis information of the data to be migrated, the original system information being used to represent the data architecture of the original system, the target system information being used to represent the data architecture of the target system, and the data analysis information being information generated after an analysis tool analyzes the data to be migrated in the original system; A determination module, the determination module being used to determine a data migration strategy for the data to be migrated based on the original system information, the target system information and the data analysis information; A migration module, wherein the migration module is used to migrate the data to be migrated in the original system to the target system according to the data migration strategy.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data migration method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the computer-readable storage medium is controlled, and the device where the program is located executes the data migration method according to any one of claims 1 to 6.

10. A processor, characterized in that: The processor is used to run a program, wherein the program executes the data migration method described in any one of claims 1 to 6 when running.

Images