Data migration methods, apparatus, terminal equipment, and storage media

Abstract

This invention discloses a data migration method, apparatus, terminal device, and storage medium, relating to the field of database technology. The method includes: acquiring data to be migrated from a first database; and migrating the data to be migrated from the first database to a second database using a pre-created data migration model, wherein the data migration model is created based on database resources and application resources. This invention can efficiently utilize application resources and database resources.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a data migration method, apparatus, terminal device, and storage medium. Background Technology

[0002] As businesses grow and their business needs change, databases, as a core component for data storage and management, play a crucial role. However, during business operations, situations may arise such as business growth and expansion, technological upgrades and innovations, data center optimization, or cloud migration. In these cases, data migration technology is needed to help businesses migrate their data. Because enterprise databases typically have large tables, numerous records, and complex table mapping rules, current data migration solutions couple data reading, data mapping, and data writing tasks into a single task.

[0003] However, current data migration solutions have shortcomings: since most enterprises have limited application and database resources, coupling data reading, data mapping, and data writing tasks into a single task does not divide the task according to the required resources (database resources and application resources), thus failing to efficiently utilize application and database resources. Summary of the Invention

[0004] The main objective of this invention is to provide a data migration method, apparatus, terminal device, and storage medium, aiming to solve the problem that current data migration technologies cannot efficiently utilize application resources and database resources.

[0005] To achieve the above objectives, the present invention provides a data migration method, the method comprising:

[0006] Retrieve the data to be migrated from the first database;

[0007] Data to be migrated from the first database to the second database is migrated using a pre-created data migration model, which is based on database resources and application resources.

[0008] Optionally, the step of migrating the data to be migrated from the first database to the second database using a pre-created data migration model includes:

[0009] The data migration model is created based on the database resources and the application resources.

[0010] Optionally, the step of creating a data migration model based on the database resources and the application resources includes:

[0011] A task node set is created based on the database resources and the application resources. Each task node in the task node set contains corresponding input, output, and processing logic.

[0012] Create a buffer queue;

[0013] The data migration model is obtained by connecting the task node set and the buffer queue.

[0014] Optionally, the step of creating the buffer queue is followed by:

[0015] Set the producer-consumer pattern for the buffer queue.

[0016] Optionally, the step of creating a task node set based on the database resources and the application resources includes:

[0017] A first task node set is created based on the resources of the first database. The first task node set is used for batch data reading, and the task nodes in the first task node set run concurrently.

[0018] A second task node set is created based on the application resources. The second task node set is used for data mapping, and the task nodes in the second task node set run concurrently.

[0019] A third task node set is created based on the resources of the second database. The third task node set is used for batch data writing, and the task nodes in the third task node set run concurrently.

[0020] Optionally, the step of creating the buffer queue includes:

[0021] Create a first buffer queue and a second buffer queue.

[0022] Optionally, the step of connecting the task node set and the buffer queue to obtain the data migration model includes:

[0023] Connect the first buffer queue to the first task node set and the second task node set, and connect the second buffer queue to the second task node set and the third task node set to obtain the data migration model.

[0024] Optionally, the step of connecting the first buffer queue to the first task node set and the second task node set, and connecting the second buffer queue to the second task node set and the third task node set, includes:

[0025] The output of the first task node set is received through the first buffer queue;

[0026] Input data is provided to the second task node set through the first buffer queue;

[0027] The output of the second task node set is received through the second buffer queue;

[0028] Input data is provided to the third task node set through the second buffer queue.

[0029] This invention also proposes a data migration apparatus, the apparatus comprising:

[0030] The data acquisition device acquires the data to be migrated from the first database;

[0031] The data processing device migrates data to be migrated from the first database to the second database using a pre-created data migration model, which is created based on database resources and application resources.

[0032] This invention also proposes a terminal device, which includes a memory, a processor, and a data migration program stored in the memory and executable on the processor. When the data migration program is executed by the processor, it implements the data migration method described above.

[0033] This invention also proposes a computer-readable storage medium storing a data migration program, which, when executed by a processor, implements the data migration method described above.

[0034] The data migration method, apparatus, terminal device, and storage medium proposed in this invention acquire data to be migrated from a first database; and migrate the data to be migrated from the first database to a second database using a pre-created data migration model, which is created based on database resources and application resources. This invention migrates data from the first database to the second database using a data migration model created based on database resources and application resources, using database resources and application resources as conditions for task partitioning and concurrency settings, thus enabling efficient and rational utilization of database resources and application resources. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the functional modules of the terminal device to which the data migration device of the present invention belongs;

[0036] Figure 2 This is a flowchart illustrating an exemplary embodiment of the data migration method of the present invention;

[0037] Figure 3 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention;

[0038] Figure 4 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention;

[0039] Figure 5a 5b and 5c are schematic diagrams of data migration models in different construction processes of the present invention;

[0040] Figure 6 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention;

[0041] Figure 7 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention;

[0042] Figure 8 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention;

[0043] Figure 9 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention.

[0044] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0045] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0046] The main solution of this invention is: to obtain the data to be migrated from a first database; and to migrate the data to be migrated from the first database to a second database using a pre-created data migration model, wherein the data migration model is created based on database resources and application resources. This invention migrates the data to be migrated from the first database to the second database using a data migration model created based on database resources and application resources. By using database resources and application resources as conditions for task partitioning and setting concurrency, it can efficiently and rationally utilize database resources and application resources.

[0047] Technical terms involved in the embodiments of this invention:

[0048] Database data migration: Database data migration refers to the process of migrating data from one database to another database system or to a different location or structure within the same database system.

[0049] Data mapping: Data mapping is an important step in the database migration process. Data mapping refers to mapping the data structure, format, and type in the source database to the corresponding structure and format supported by the target database.

[0050] Producer-Consumer Pattern: The producer-consumer pattern is a concurrent programming pattern used to solve the problem of data interaction and collaboration between producers and consumers. This pattern is typically used in multi-threaded or multi-process environments, where producers are responsible for generating data and placing it into a shared buffer, while consumers retrieve data from the buffer and process it.

[0051] Data-intensive tasks: Data-intensive tasks refer to those that require significant computing and storage resources when processing large amounts of data. These tasks typically involve reading, writing, computing, and analyzing large-scale data, requiring efficient utilization of database and application resources.

[0052] The embodiments of the present invention take into account the following: Current data migration technology solutions have defects: Since most enterprises have limited application resources and database resources, the practice of coupling data reading, data mapping and data writing tasks into one task does not divide the task according to the required resources (database resources, application resources), thus failing to efficiently utilize application resources and database resources.

[0053] Therefore, this invention proposes a solution that uses a data migration model created based on database resources and application resources to migrate data to be migrated from the first database to the second database. By using database resources and application resources as conditions for task partitioning and setting concurrency, database resources and application resources can be utilized efficiently and rationally.

[0054] Specifically, refer to Figure 1 , Figure 1 This is a functional module diagram of the terminal device to which the data migration device of the present invention belongs. The data migration device can be a data processing device independent of the device itself, which can be carried on the device in hardware or software form. The device can be a smart mobile terminal with data processing capabilities, such as a mobile phone or tablet computer, or it can be a fixed device or server with data processing capabilities.

[0055] In this embodiment, the data migration device includes at least an output module 110, a processor 120, a memory 130, and a communication module 140.

[0056] The memory 130 stores the operating system and data migration program; the output module 110 may be a display screen, etc. The communication module 140 may include a WIFI module and a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0057] When the data migration program in memory 130 is executed by the processor, it performs the following steps:

[0058] Retrieve the data to be migrated from the first database;

[0059] Data to be migrated from the first database to the second database is migrated using a pre-created data migration model, which is based on database resources and application resources.

[0060] Furthermore, when the data migration program in memory 130 is executed by the processor, it also performs the following steps:

[0061] The data migration model is created based on the database resources and the application resources.

[0062] Furthermore, when the data migration program in memory 130 is executed by the processor, it also performs the following steps:

[0063] A task node set is created based on the database resources and the application resources. Each task node in the task node set contains corresponding input, output, and processing logic.

[0064] Create a buffer queue;

[0065] The data migration model is obtained by connecting the task node set and the buffer queue.

[0066] Furthermore, when the data migration program in memory 130 is executed by the processor, it also performs the following steps:

[0067] Set the producer-consumer pattern for the buffer queue.

[0068] Furthermore, when the data migration program in memory 130 is executed by the processor, it also performs the following steps:

[0069] A first task node set is created based on the resources of the first database. The first task node set is used for batch data reading, and the task nodes in the first task node set run concurrently.

[0070] A second task node set is created based on the application resources. The second task node set is used for data mapping, and the task nodes in the second task node set run concurrently.

[0071] A third task node set is created based on the resources of the second database. The third task node set is used for batch data writing, and the task nodes in the third task node set run concurrently.

[0072] Furthermore, when the data migration program in memory 130 is executed by the processor, it also performs the following steps:

[0073] Create a first buffer queue and a second buffer queue;

[0074] Connect the first buffer queue to the first task node set and the second task node set, and connect the second buffer queue to the second task node set and the third task node set to obtain the data migration model.

[0075] Furthermore, when the data migration program in memory 130 is executed by the processor, it also performs the following steps:

[0076] The output of the first task node set is received through the first buffer queue;

[0077] Input data is provided to the second task node set through the first buffer queue;

[0078] The output of the second task node set is received through the second buffer queue;

[0079] Input data is provided to the third task node set through the second buffer queue.

[0080] This embodiment, through the above-described scheme, specifically obtains the data to be migrated from the first database; and migrates the data to be migrated from the first database to the second database using a pre-created data migration model, which is created based on database resources and application resources. This embodiment of the invention migrates the data to be migrated from the first database to the second database using a data migration model created based on database resources and application resources. By using database resources and application resources as conditions for task partitioning and setting concurrency, it can efficiently and rationally utilize database resources and application resources.

[0081] Based on, but not limited to, the above-described device architecture, embodiments of the method of the present invention are proposed.

[0082] The execution subject of the method in this embodiment can be a data migration device, which can be a device that is independent of the device and capable of data processing, and can be carried on the device in the form of hardware or software.

[0083] Reference Figure 2 , Figure 2 This is a flowchart illustrating an exemplary embodiment of the data migration method of the present invention. The data migration method includes:

[0084] Step S20: Obtain the data to be migrated from the first database.

[0085] One implementation method is to use appropriate database query statements to obtain the data to be migrated from the first database.

[0086] Step S30: The data to be migrated from the first database to the second database is migrated using a pre-created data migration model, which is created based on database resources and application resources.

[0087] In the course of business operations, situations may arise such as business growth and expansion, technology upgrades and innovations, data center optimization, or cloud migration. In these cases, data migration technology is needed to help enterprises migrate their data.

[0088] The data migration model proposed in this invention is created based on database resources and application resources. In scenarios where database resources and application resources are limited, the data migration model proposed in this invention can make efficient and reasonable use of database resources and application resources.

[0089] Database resources refer to the hardware and software resources associated with a database system. These include, but are not limited to, database servers, storage devices, network connections, database engines, and management tools. The performance and reliability of database resources are crucial to the efficiency and success of data migration. During data migration, database resources need to be configured and managed appropriately to ensure that the database system can handle and store the migrated data while meeting performance and reliability requirements.

[0090] Application resources refer to the resources required by the applications and environment related to data migration. This includes the application's runtime environment, computing resources, memory, network bandwidth, etc. During data migration, applications may be needed to connect to and manipulate the source and target databases, and perform specific data migration operations (such as data transformation, mapping, etc.). To ensure a smooth data migration, application resources need to be assessed and managed to meet the application's operational requirements.

[0091] This embodiment, through the above-described scheme, obtains the data to be migrated from the first database; and migrates the data to be migrated from the first database to the second database using a pre-created data migration model. This data migration model is created based on database resources and application resources. This embodiment of the invention migrates the data to be migrated from the first database to the second database using a data migration model created based on database resources and application resources. By using database resources and application resources as conditions for task partitioning and setting concurrency, it can efficiently and rationally utilize database resources and application resources.

[0092] Reference Figure 3 , Figure 3 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention.

[0093] Based on the above Figure 2In the embodiment shown, step S20, before obtaining the data to be migrated from the first database, includes:

[0094] Step S10: Create the data migration model based on the database resources and the application resources.

[0095] One implementation method involves creating a task node set based on database resources and application resources. Each task node in the task node set contains corresponding input, output, and processing logic. Then, a buffer queue is created, and the task node set and the buffer queue are connected to obtain a data migration model.

[0096] This embodiment, through the above-described scheme, obtains the data to be migrated from a first database; and migrates the data to be migrated from the first database to a second database using a pre-created data migration model. This data migration model is created based on database resources and application resources. Specifically, the data migration model is created based on the database resources and the application resources.

[0097] This invention, through a data migration model created based on database and application resources, migrates data from a first database to a second database. By using database and application resources as conditions for task partitioning and concurrency settings, it achieves efficient and rational utilization of these resources. Specifically, this invention, by creating a data migration model based on database and application resources, provides a model for data migration within a database that features reasonable task partitioning, appropriate concurrency, and efficient scheduling.

[0098] Reference Figure 4 , Figure 4 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention.

[0099] Based on the above Figure 3 In the illustrated embodiment, step S10, creating the data migration model based on the database resources and the application resources, includes:

[0100] Step S11: Create a task node set based on the database resources and the application resources. Each task node in the task node set contains corresponding input, output, and processing logic.

[0101] The task node set consists of one or more task nodes, each of which contains corresponding input, output, and processing logic.

[0102] The input to a task node can be data from a database or the output of the previous task node; the output of a task node is the processing result of the current task node, which can be stored in a database or passed to the next task node; and the processing logic of a task node is the core of the task node, defining the specific data processing methods and algorithms.

[0103] In this process, task node sets with different functions can be created based on the data migration steps.

[0104] In one implementation method, the data migration steps can be batch data reading and batch data writing, so two task node sets can be created for batch data reading and batch data writing respectively.

[0105] In another implementation, the data migration steps can also be batch data reading, data mapping, and batch data writing. Therefore, three task node sets can be created for batch data reading, data mapping, and batch data writing, respectively.

[0106] In each function's task node set, the task nodes are executed concurrently.

[0107] The number of task nodes varies depending on the function of the task node set, and the specific number needs to be set according to the database resources or application resources.

[0108] Step S12: Create a buffer queue.

[0109] The buffer queue is a data structure that can serve as an intermediate buffer area. It should be noted that this invention is not limited to using buffer queues; any structure that can serve as an intermediate buffer area can also be used.

[0110] Among them, the buffer queue is used for data transfer between task node sets, so a certain number of buffer queues can be set according to the number of task node sets. The capacity of different buffer queues is set according to the actual situation.

[0111] One implementation approach is to adjust the buffer queue capacity based on the difference between the input and output speeds during actual tasks or testing. If the input buffer queue is faster than the output buffer queue, the buffer queue needs to be large enough to store the data to be processed and prevent data overflow. If the output buffer queue is faster than the input buffer queue, the buffer queue capacity can be appropriately reduced.

[0112] In one implementation approach, the capacity of the buffer queue needs to take into account resource limitations, such as available memory space. If resources are limited, a reasonable buffer queue capacity needs to be set according to the actual situation to avoid resource exhaustion and other problems.

[0113] Step S14: Connect the task node set and the buffer queue to obtain the data migration model.

[0114] The buffer queue is responsible for connecting task node sets, receiving the output of the previous task node set, and serving as the input of the next task node set.

[0115] Reference Figure 5a ,5b,5c, Figure 5a Figures 5b and 5c are schematic diagrams of data migration models in different construction processes of this invention.

[0116] First, assuming each task node set has only one task node, we connect the buffer queues of numerous task node sets to obtain a pipeline model, such as... Figure 5a As shown.

[0117] Then, assuming that each task node set has 3 task nodes and each buffer queue has a capacity of 10, we can obtain the following: Figure 5b The model shown.

[0118] Finally, considering the data migration scenario, where both the data input and output scenarios involve databases, we can obtain the following... Figure 5c The model shown.

[0119] This embodiment, through the above-described scheme, obtains data to be migrated from a first database; and migrates the data to be migrated from the first database to a second database using a pre-created data migration model, which is created based on database resources and application resources. Specifically, the data migration model is created based on the database resources and application resources. Furthermore, a task node set is created based on the database resources and application resources, and each task node in the task node set contains corresponding input, output, and processing logic; a buffer queue is created; and the task node set and the buffer queue are connected to obtain the data migration model.

[0120] This invention, through a data migration model created based on database and application resources, migrates data from a first database to a second database. Using database and application resources as conditions for task partitioning and concurrency settings, it efficiently and rationally utilizes these resources. Specifically, this invention provides a model for data migration in the database that rationally partitions tasks, implements appropriate concurrency, and efficiently schedules processes. Furthermore, by connecting the task node set to a buffer queue, task input and output can be asynchronous. This better utilizes database and application resources, avoids resource waste and blocking, and improves resource utilization. Moreover, by using the data migration model, this invention allows for the addition or deletion of task nodes as needed, and the adjustment of task processing logic and flow, thereby improving the model's scalability and flexibility to meet diverse business needs.

[0121] Reference Figure 6 , Figure 6 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention.

[0122] Based on the above Figure 4 In the embodiment shown, step S12, after creating the buffer queue, includes:

[0123] Step S13: Set the producer-consumer mode for the buffer queue.

[0124] In one implementation approach, the producer role is first defined: within the buffer queue, the producer is responsible for generating data and placing it into the queue. The producer can be one or more task nodes that acquire data from the input source, process it, generate output data, and then place it into the buffer queue.

[0125] Next, define the consumer role: In the buffer queue, the consumer is responsible for retrieving data from the queue and processing it. Consumers can be one or more other task nodes that retrieve data from the buffer queue and perform corresponding processing operations, such as further computation, analysis, or storage.

[0126] Next, a synchronization mechanism needs to be set up: To ensure coordinated operation between producers and consumers, an appropriate synchronization mechanism is required. For example, mutexes or semaphores can be used to control access to the buffer queue, preventing data races or errors caused by multiple producers and consumers reading and writing simultaneously.

[0127] In this process, after producing data, the producer first checks the status of the buffer queue to ensure there is enough space to hold the data. If the queue is full, the producer must wait until space becomes available. Once space is available, the producer adds the data to the queue and notifies the consumer that processing can begin. Before processing data, the consumer checks the status of the buffer queue to ensure there is data available for processing. If the queue is empty, the consumer must wait until data becomes available. Once data is available in the queue, the consumer retrieves the data, processes it, and notifies the producer that data production can continue.

[0128] This embodiment, through the above-described scheme, obtains data to be migrated from a first database; and migrates the data to be migrated from the first database to a second database using a pre-created data migration model, which is created based on database resources and application resources. Specifically, the data migration model is created based on the database resources and application resources. A task node set is created based on the database resources and application resources, and each task node in the task node set contains corresponding input, output, and processing logic; a buffer queue is created; and the task node set and the buffer queue are connected to obtain the data migration model. A producer-consumer pattern is set for the buffer queue.

[0129] This invention, through a data migration model created based on database and application resources, migrates data from a first database to a second database. Using database and application resources as conditions for task partitioning and concurrency settings, it efficiently and rationally utilizes these resources. Specifically, this invention provides a model for data migration in the database that rationally partitions tasks, implements reasonable concurrency, and efficiently schedules processes. Furthermore, by connecting the task node set to a buffer queue, task input and output can be asynchronous. This better utilizes database and application resources, avoids resource waste and blocking, and improves resource utilization. Moreover, by using the data migration model, this invention allows for the addition or deletion of task nodes as needed, adjusting task processing logic and flow, thereby improving the model's scalability and flexibility to meet different business needs. Finally, by setting a producer-consumer pattern for the buffer queue, producers can continuously send data to the buffer queue, while consumers can retrieve data from the buffer queue for processing, achieving asynchronous processing and parallel computing. This accelerates task execution and improves system response performance.

[0130] Reference Figure 7 , Figure 7 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention.

[0131] Based on the above Figure 6 In the illustrated embodiment, step S11, creating a task node set based on the database resources and the application resources, wherein each task node in the task node set includes corresponding input, output, and processing logic, including:

[0132] Step S111: Create a first task node set based on the resources of the first database. The first task node set is used for batch data reading, and the task nodes in the first task node set run concurrently.

[0133] Among them, the appropriate number of concurrent task nodes in the first task node cluster is determined by comprehensively considering the database performance, system load, hardware configuration, network bandwidth and actual business needs of the first database.

[0134] One implementation method allows for flexible adjustment of the concurrency level to meet actual needs, and monitoring of system performance and resource utilization to ensure efficient operation of the task node set.

[0135] The first task node set is used to read data from the first database in batches.

[0136] Step S112: Create a second task node set based on the application resources. The second task node set is used for data mapping, and the task nodes in the second task node set run concurrently.

[0137] In the process of database data migration, since there may be differences in the data models, structures, types and values ​​between the source database and the target database, data mapping can solve the problem of differences between the source database and the target database, ensuring that the data in the source database can be correctly and completely migrated to the target database.

[0138] In one implementation, data mapping can be performed on the backend or server side of the application, where the backend or server side has application resources.

[0139] Among them, the appropriate number of concurrent task nodes in the second task node set is determined by comprehensively considering the application resources such as memory resources, CPU resources, disk resources, network resources and actual business needs.

[0140] One implementation method allows for flexible adjustment of the concurrency level to meet actual needs, and monitoring of system performance and resource utilization to ensure efficient operation of the task node set.

[0141] Step S113: Create a third task node set based on the resources of the second database. The third task node set is used for batch data writing, and the task nodes in the third task node set run concurrently.

[0142] Among them, the appropriate number of concurrent tasks for the third task node is determined by comprehensively considering the database performance, system load, hardware configuration, network bandwidth and actual business needs of the second database.

[0143] One implementation method allows for flexible adjustment of the concurrency level to meet actual needs, and monitoring of system performance and resource utilization to ensure efficient operation of the task node set.

[0144] The third task node set is used to write data in batches into the second database.

[0145] This embodiment, through the above-described scheme, obtains data to be migrated from a first database; migrates the data to be migrated from the first database to a second database using a pre-created data migration model, which is created based on database resources and application resources. Specifically, the data migration model is created based on the database resources and application resources. A task node set is created based on the database resources and application resources, each task node in the task node set containing corresponding input, output, and processing logic; a buffer queue is created; and the task node set and the buffer queue are connected to obtain the data migration model. A producer-consumer pattern is set for the buffer queue. Further, a first task node set is created based on the resources of the first database, used for batch data reading, and the task nodes in the first task node set run concurrently; a second task node set is created based on the application resources, used for data mapping, and the task nodes in the second task node set run concurrently; a third task node set is created based on the resources of the second database, used for batch data writing, and the task nodes in the third task node set run concurrently.

[0146] This invention, through a data migration model created based on database and application resources, migrates data from a first database to a second database. Using database and application resources as conditions for task partitioning and concurrency settings, it efficiently and rationally utilizes these resources. Specifically, this invention provides a model for data migration in the database that rationally partitions tasks, implements reasonable concurrency, and efficiently schedules processes. Furthermore, by connecting the task node set to a buffer queue, task input and output can be asynchronous. This better utilizes database and application resources, avoids resource waste and blocking, and improves resource utilization. Moreover, by using the data migration model, this invention allows for the addition or deletion of task nodes as needed, adjusting task processing logic and flow, thereby improving the model's scalability and flexibility to meet different business needs. Finally, by setting a producer-consumer pattern for the buffer queue, producers can continuously send data to the buffer queue, while consumers can retrieve data from the buffer queue for processing, achieving asynchronous processing and parallel computing. This accelerates task execution and improves system response performance. Furthermore, in scenarios with limited application and database resources, this embodiment of the invention divides the database migration process into several independent steps during data-intensive data processing: batch data reading, data mapping, and batch data writing. Each step is configured with a reasonable number of concurrent task nodes and a reasonable buffer queue capacity based on available resources, thereby making full use of resources and improving data processing efficiency.

[0147] Reference Figure 8 , Figure 8 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention.

[0148] Based on the above Figure 7 In the embodiment shown, step S12, creating the buffer queue, includes:

[0149] Step S121: Create the first buffer queue and the second buffer queue.

[0150] One implementation method involves setting the capacity of the first and second buffer queues based on the difference between the input and output speeds of the buffer queues during actual tasks or tests. If the input buffer queue is faster than the output buffer queue, the buffer queues need to be large enough to store the data to be processed and prevent data overflow. If the output buffer queue is faster than the input buffer queue, the capacity of the buffer queues can be appropriately reduced.

[0151] In one implementation approach, the capacity of the first and second buffer queues needs to take into account resource limitations, such as available memory space. If resources are limited, then the capacity of the first and second buffer queues needs to be set reasonably according to the actual situation to avoid resource exhaustion and other problems.

[0152] Based on the above Figure 7 In the embodiment shown, step S14, which connects the task node set and the buffer queue to obtain the data migration model, includes:

[0153] Step S141: Connect the first buffer queue to the first task node set and the second task node set, and connect the second buffer queue to the second task node set and the third task node set to obtain the data migration model.

[0154] The first buffer queue receives the output of the first task node set and serves as the input of the second task node set, while the second buffer queue receives the output of the second task node set and serves as the input of the third task node set.

[0155] This embodiment, through the above-described scheme, obtains data to be migrated from a first database; migrates the data to be migrated from the first database to a second database using a pre-created data migration model, which is created based on database resources and application resources. Specifically, the data migration model is created based on the database resources and application resources. A task node set is created based on the database resources and application resources, each task node in the task node set containing corresponding input, output, and processing logic; a buffer queue is created; and the task node set and the buffer queue are connected to obtain the data migration model. A producer-consumer pattern is set for the buffer queue. Further, a first task node set is created based on the resources of the first database, used for batch data reading, and the task nodes in the first task node set run concurrently; a second task node set is created based on the application resources, used for data mapping, and the task nodes in the second task node set run concurrently; a third task node set is created based on the resources of the second database, used for batch data writing, and the task nodes in the third task node set run concurrently. Further, a first buffer queue and a second buffer queue are created; the first buffer queue is connected to the first task node set and the second task node set, and the second buffer queue is connected to the second task node set and the third task node set to obtain a data migration model.

[0156] This invention, through a data migration model created based on database and application resources, migrates data from a first database to a second database. Using database and application resources as conditions for task partitioning and concurrency settings, it efficiently and rationally utilizes these resources. Specifically, this invention provides a model for data migration in the database that rationally partitions tasks, implements reasonable concurrency, and efficiently schedules processes. Furthermore, by connecting the task node set to a buffer queue, task input and output can be asynchronous. This better utilizes database and application resources, avoids resource waste and blocking, and improves resource utilization. Moreover, by using the data migration model, this invention allows for the addition or deletion of task nodes as needed, adjusting task processing logic and flow, thereby improving the model's scalability and flexibility to meet different business needs. Finally, by setting a producer-consumer pattern for the buffer queue, producers can continuously send data to the buffer queue, while consumers can retrieve data from the buffer queue for processing, achieving asynchronous processing and parallel computing. This accelerates task execution and improves system response performance. Furthermore, in scenarios with limited application and database resources, this embodiment of the invention divides the database migration process into several independent steps during data-intensive data processing: batch data reading, data mapping, and batch data writing. Each step is configured with a reasonable number of concurrent task nodes and a reasonable buffer queue capacity based on available resources, thereby fully utilizing resources and improving data processing efficiency. Furthermore, by connecting the first buffer queue to the first and second task node sets, and connecting the second buffer queue to the second and third task node sets, this embodiment of the invention enables sequential control of the data flow, ensuring that data is processed according to the specified process order.

[0157] Reference Figure 9 , Figure 9 This is a flowchart illustrating another exemplary embodiment of the data migration method of the present invention.

[0158] Based on the above Figure 8 In the illustrated embodiment, step S141, connecting the first buffer queue to the first task node set and the second task node set, and connecting the second buffer queue to the second task node set and the third task node set, to obtain the data migration model includes:

[0159] Step S1411: Receive the output of the first task node set through the first buffer queue.

[0160] The first buffer queue receives the output from the first task node set and stores it as temporary data.

[0161] Step S1412: Input data is provided to the second task node set through the first buffer queue.

[0162] The first buffer queue passes the stored temporary data as input to the second task node set.

[0163] Step S1413: The output of the second task node set is received through the second buffer queue.

[0164] The second buffer queue receives the output from the second task node set and stores it as temporary data.

[0165] Step S1414: Input data is provided to the third task node set through the second buffer queue.

[0166] The second buffer queue stores temporary data as input and passes it to the third task node set.

[0167] This embodiment, through the above-described scheme, obtains data to be migrated from a first database; migrates the data to be migrated from the first database to a second database using a pre-created data migration model, which is created based on database resources and application resources. Specifically, the data migration model is created based on the database resources and application resources. A task node set is created based on the database resources and application resources, each task node in the task node set containing corresponding input, output, and processing logic; a buffer queue is created; and the task node set and the buffer queue are connected to obtain the data migration model. A producer-consumer pattern is set for the buffer queue. Further, a first task node set is created based on the resources of the first database, used for batch data reading, and the task nodes in the first task node set run concurrently; a second task node set is created based on the application resources, used for data mapping, and the task nodes in the second task node set run concurrently; a third task node set is created based on the resources of the second database, used for batch data writing, and the task nodes in the third task node set run concurrently. Further, a first buffer queue and a second buffer queue are created; the first buffer queue is connected to the first task node set and the second task node set, and the second buffer queue is connected to the second task node set and the third task node set to obtain a data migration model. Further, the first buffer queue receives the output of the first task node set; the first buffer queue provides input data to the second task node set; the second buffer queue receives the output of the second task node set; and the second buffer queue provides input data to the third task node set.

[0168] This invention, through a data migration model created based on database and application resources, migrates data from a first database to a second database. Using database and application resources as conditions for task partitioning and concurrency settings, it efficiently and rationally utilizes these resources. Specifically, this invention provides a model for data migration in the database that rationally partitions tasks, implements reasonable concurrency, and efficiently schedules processes. Furthermore, by connecting the task node set to a buffer queue, task input and output can be asynchronous. This better utilizes database and application resources, avoids resource waste and blocking, and improves resource utilization. Moreover, by using the data migration model, this invention allows for the addition or deletion of task nodes as needed, adjusting task processing logic and flow, thereby improving the model's scalability and flexibility to meet different business needs. Finally, by setting a producer-consumer pattern for the buffer queue, producers can continuously send data to the buffer queue, while consumers can retrieve data from the buffer queue for processing, achieving asynchronous processing and parallel computing. This accelerates task execution and improves system response performance. Furthermore, in scenarios with limited application and database resources, this embodiment of the invention divides the database migration process into several independent steps during data-intensive data processing: batch data reading, data mapping, and batch data writing. Each step is configured with a reasonable number of concurrent task nodes and a reasonable buffer queue capacity based on available resources, thereby fully utilizing resources and improving data processing efficiency. Furthermore, by connecting the first buffer queue to the first and second task node sets, and connecting the second buffer queue to the second and third task node sets, this embodiment of the invention enables sequential control of the data flow, ensuring that data is processed according to the specified process order.

[0169] Furthermore, embodiments of this application also propose a data migration apparatus, the data migration apparatus comprising:

[0170] The data acquisition device acquires the data to be migrated from the first database;

[0171] The data processing device migrates data to be migrated from the first database to the second database using a pre-created data migration model, which is created based on database resources and application resources.

[0172] The principle and implementation process of data migration in this embodiment are explained in the above embodiments and will not be repeated here.

[0173] Furthermore, this application also proposes a terminal device, which includes a memory, a processor, and a data migration program stored in the memory and executable on the processor. When the data migration program is executed by the processor, it implements the steps of the data migration method described above.

[0174] Since this data migration program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be elaborated here.

[0175] Furthermore, embodiments of this application also propose a computer-readable storage medium storing a data migration program, which, when executed by a processor, implements the steps of the data migration method described above.

[0176] Since this data migration program employs all the technical solutions of all the foregoing embodiments when executed by the processor, it has at least all the beneficial effects brought about by all the technical solutions of all the foregoing embodiments, which will not be elaborated here.

[0177] This embodiment, through the above-described scheme, specifically obtains the data to be migrated from the first database; and migrates the data to be migrated from the first database to the second database using a pre-created data migration model, which is created based on database resources and application resources. This embodiment of the invention migrates the data to be migrated from the first database to the second database using a data migration model created based on database resources and application resources. By using database resources and application resources as conditions for task partitioning and setting concurrency, it can efficiently and rationally utilize database resources and application resources.

[0178] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or approach that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or approach. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or approach that includes that element.

[0179] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of the present invention.

[0181] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A data migration method, characterized in that, The method includes the following steps: Retrieve the data to be migrated from the first database; A task node set is created based on database resources and application resources. Each task node in the task node set contains corresponding input, output, and processing logic. Create a buffer queue; Set the buffer queue to a producer-consumer pattern; By connecting the task node set and the buffer queue, a data migration model is obtained; The data migration model is used to migrate data from the first database to the second database. The data migration model is created based on database resources and application resources. The step of creating a task node set based on the database resources and the application resources includes: A first task node set is created based on the resources of the first database. The first task node set is used for batch data reading, and the task nodes in the first task node set run concurrently. A second task node set is created based on the application resources. The second task node set is used for data mapping, and the task nodes in the second task node set run concurrently. A third task node set is created based on the resources of the second database. The third task node set is used for batch data writing, and the task nodes in the third task node set run concurrently.

2. The method according to claim 1, characterized in that, The steps for creating the buffer queue include: Create a first buffer queue and a second buffer queue; The step of connecting the task node set and the buffer queue to obtain the data migration model includes: The first buffer queue is connected to the first task node set and the second task node set, and the second buffer queue is connected to the second task node set and the third task node set to obtain the data migration model.

3. The method according to claim 2, characterized in that, The steps of connecting the first buffer queue to the first task node set and the second task node set, and connecting the second buffer queue to the second task node set and the third task node set, include: The output of the first task node set is received through the first buffer queue; Input data is provided to the second task node set through the first buffer queue; The output of the second task node set is received through the second buffer queue; Input data is provided to the third task node set through the second buffer queue.

4. A data migration device, characterized in that, The device includes: The data acquisition module is used to acquire the data to be migrated from the first database; The data acquisition module is also used to create a task node set based on database resources and application resources, wherein each task node in the task node set contains corresponding input, output, and processing logic; create a buffer queue; set a producer-consumer pattern for the buffer queue; and connect the task node set and the buffer queue to obtain a data migration model. The data processing module is used to migrate data to be migrated from the first database to the second database through the data migration model, wherein the data migration model is created based on database resources and application resources; The data processing module is further configured to create a first task node set based on the resources of the first database, the first task node set being used for batch data reading, and the task nodes in the first task node set running concurrently; create a second task node set based on the application resources, the second task node set being used for data mapping, and the task nodes in the second task node set running concurrently; and create a third task node set based on the resources of the second database, the third task node set being used for batch data writing, and the task nodes in the third task node set running concurrently.

5. A data migration terminal device, characterized in that, The data migration terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the data migration method as described in any one of claims 1-3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the data migration method as described in any one of claims 1-3.

Images