Data migration method, system and device

Abstract

The embodiment of the invention provides a data migration method, system and device, and the method comprises the steps: determining a to-be-migrated data stream in a source database, and migrating the to-be-migrated data stream from the source database to a target database through an initial memory space, and the initial memory space is determined according to a memory distribution strategy; detecting whether the utilization rate of the initial memory space meets a memory use condition or not in a data migration process; if yes, releasing a to-be-released memory space in the initial memory space, and continuing to perform data migration on the to-be-migrated data stream by using an unreleased memory space in the initial memory space; and if not, expanding the initial memory space into a target memory space according to the memory allocation information, and continuing to perform data migration on the to-be-migrated data stream by utilizing the target memory space.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to data migration methods, systems and apparatus. Background Technology

[0002] With the advent of the big data era, the amount of data that needs to be stored is growing exponentially. Database technology is constantly innovating, and the need to migrate data between databases is also increasing. When migrating data between two databases, the data to be migrated includes not only the full dataset but also incremental data generated during the migration process. Therefore, the allocation of memory resources becomes particularly important when dealing with these two types of data to be migrated.

[0003] In existing technologies, the data blocks to be migrated and the amount of data in each block are typically determined before data migration, and appropriate memory resources are allocated accordingly. However, due to the uncertainty of memory requirements during data migration, memory overflow issues can easily occur. This can force data migration to be interrupted, affecting migration efficiency. Therefore, a more effective data migration method is urgently needed to solve the above problems. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a data migration method. One or more embodiments of this specification also relate to a data migration apparatus, a data migration system, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a data migration method is provided, comprising:

[0006] The data stream to be migrated is determined in the source database, and the data stream to be migrated is migrated from the source database to the target database through an initial memory space, wherein the initial memory space is determined according to a memory allocation strategy;

[0007] During the data migration process, the utilization rate of the initial memory space is checked to see if it meets the memory usage conditions.

[0008] If so, the memory space to be released in the initial memory space is released, and the data migration of the data stream to be migrated is continued using the memory space not released in the initial memory space;

[0009] If not, the initial memory space is expanded into a target memory space according to the memory allocation information, and the target memory space is used to continue migrating the data stream to be migrated.

[0010] According to a second aspect of the embodiments of this specification, a data migration apparatus is provided, comprising:

[0011] The migration module is configured to determine the data stream to be migrated in the source database and migrate the data stream to be migrated from the source database to the target database through an initial memory space, wherein the initial memory space is determined according to a memory allocation strategy;

[0012] The judgment module is configured to detect whether the utilization rate of the initial memory space meets the memory usage conditions during the data migration process;

[0013] If so, the release module is run. The release module is configured to release the memory space to be released in the initial memory space, and use the unreleased memory space in the initial memory space to continue to migrate the data stream to be migrated.

[0014] If not, the expansion module is run. The expansion module is configured to expand the initial memory space into a target memory space according to the memory allocation information, and use the target memory space to continue to migrate the data stream to be migrated.

[0015] According to a third aspect of the embodiments of this specification, a data migration system is provided, including a memory management node, a source data processing node corresponding to a source database, and a target data processing node corresponding to a target database.

[0016] The source data processing node is used to determine the data stream to be migrated in the source database and migrate the data stream to be migrated to the initial memory space, wherein the initial memory space is determined according to a memory allocation strategy;

[0017] The memory management node is used to detect whether the utilization rate of the initial memory space meets the memory usage conditions during the data migration process; if so, it releases the memory space to be released in the initial memory space and continues to migrate the data stream to be migrated using the unreleased memory space in the initial memory space; if not, it expands the initial memory space into a target memory space according to the memory allocation information and continues to migrate the data stream to be migrated using the target memory space.

[0018] The target data processing node is used to send feedback information indicating that the data migration is complete to the source data processing node when the data stream to be migrated is completed.

[0019] According to a fourth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0020] Memory and processor;

[0021] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data migration method.

[0022] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data migration method described above.

[0023] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions that, when executed by a processor, implement the steps of the data migration method described above.

[0024] One embodiment of this specification describes a process for migrating data from a source database to a target database. The data is abstracted into a data stream and migrated to the target database. Based on the initial memory usage of the data stream during the migration process, the size of the initial memory space is dynamically adjusted. Specifically, if the initial memory space is sufficient, unused memory is released; if the initial memory space is insufficient, it is expanded. This dynamic adjustment of memory space according to the memory usage requirements during data migration ensures full and rational utilization of memory, effectively increases migration concurrency, and improves data migration efficiency. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the processing procedure of a data migration method provided in one embodiment of this specification;

[0026] Figure 2 This is a flowchart illustrating a data migration method provided in one embodiment of this specification;

[0027] Figure 3 This is a schematic diagram of memory space allocation for a data migration method provided in one embodiment of this specification;

[0028] Figure 4 This is a flowchart illustrating the processing procedure of a data migration method provided in one embodiment of this specification;

[0029] Figure 5 This is a flowchart illustrating a data migration method provided in one embodiment of this specification;

[0030] Figure 6 This is a flowchart illustrating the migration logic of a data migration method provided in one embodiment of this specification.

[0031] Figure 7This is a schematic diagram of the structure of a data migration device provided in one embodiment of this specification;

[0032] Figure 8 This is a structural diagram of a data migration system provided in one embodiment of this specification;

[0033] Figure 9 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0034] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0035] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0037] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0038] Figure 1 This is a schematic diagram illustrating the processing procedure of a data migration method provided in one embodiment of this specification; as shown below. Figure 1As shown, in a data migration scenario, data from the source database is temporarily stored in memory space, and then the data in memory space is migrated to the target database. The data tables to be migrated are identified in the source database, and a migration data stream is constructed based on these tables. This data stream is then migrated from the source database to the target database using the initial memory space. The initial memory space is determined based on the data volume of the migration data stream and its corresponding memory allocation information. The source database contains at least one data table to be migrated, and each table can be used to construct a corresponding migration data stream. Based on the proportion of the data volume corresponding to each migration data stream within the total data volume of the source database, the initial memory space information for each migration data stream, as well as the memory space allocation limits for each migration data stream, are determined. The initial memory space information and memory space limits for each migration data stream constitute the memory allocation information.

[0039] During the data migration process, it is checked whether the utilization rate of the initial memory space meets the memory usage conditions. If the utilization rate of the initial memory space meets the memory usage conditions, it means that the initial memory space is sufficient to store the data stream to be migrated. The memory space that has not been used for a long time in the initial memory space can be released, and the data migration of the data stream to be migrated can continue using the unreleased memory space in the initial memory space. If the utilization rate of the initial memory space does not meet the memory usage conditions, it means that the initial memory space is insufficient to store the data stream to be migrated. In this case, the initial memory space is expanded into the target memory space according to the memory allocation information, and the data migration of the data stream to be migrated can continue using the target memory space.

[0040] In summary, during the migration of data from the source database to the target database, the data is abstracted into a data stream and migrated to the target database. Based on the initial memory usage of the data stream during the migration process, the size of the initial memory space is dynamically adjusted. Specifically, if the initial memory space is sufficient, unused memory is released; if the initial memory space is insufficient, it is expanded. This dynamic adjustment of memory space according to the memory usage requirements during data migration ensures full and rational utilization of memory, effectively increases migration concurrency, and improves data migration efficiency.

[0041] This specification provides a data migration method, and also relates to a data migration apparatus, a data migration system, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0042] See Figure 2 , Figure 2A flowchart of a data migration method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0043] Step 202: Determine the data stream to be migrated in the source database, and migrate the data stream to be migrated from the source database to the target database through the initial memory space, wherein the initial memory space is determined according to the memory allocation strategy.

[0044] Specifically, the source database refers to the database from which data is migrated during the data migration process; correspondingly, the target database is the database from which data is written during the data migration process, migrating the data stored in the source database to the target database; the data stream to be migrated refers to the data stream determined in the source database that consists of data to be migrated; the data stream can consist of data from at least one table in the source database, or it can consist of data blocks in the source database; the source database contains at least one data stream to be migrated; the initial memory space is used to temporarily store the migrated data stream after it is migrated from the source database; the initial memory space can be determined in the global memory space corresponding to the source database; the memory allocation strategy is the allocation method for the initial memory space of the data stream to be migrated, which can be determined based on the data stream to be migrated, the amount of data stored in the source database, and the data distribution method of the source database.

[0045] Based on this, when migrating data from the source database to the target database, the data tables to be migrated are identified in the source database, and a data stream to be migrated is constructed based on these tables. The source database contains at least one data table to be migrated, and a data stream to be migrated can be constructed based on each table. Initial memory space is allocated to the data stream to be migrated according to a memory allocation strategy, and the data stream is migrated from the source database to the target database through the initial memory space corresponding to the data to be migrated. The data stream to be migrated is then removed from the source database and temporarily stored in the initial memory space. If the migration conditions are met, the data stream temporarily stored in the initial memory space is removed from the initial memory space and stored in the target database.

[0046] Furthermore, the source database contains at least one data stream to be migrated. At least one data stream can be migrated synchronously. Considering limited memory space, the rationality of resource allocation needs to be considered when allocating memory space for each data stream to be migrated. The specific implementation is as follows:

[0047] The amount of data to be migrated in the data stream to be migrated is determined, and the occupancy rate of the data to be migrated in the global data volume corresponding to the source database is determined; based on the occupancy rate and the global memory information corresponding to the source database, the local memory information corresponding to the data stream to be migrated is determined; the initial memory information is determined according to the amount of data to be migrated, and the initial memory information and the local memory information are used as the memory allocation information corresponding to the data stream to be migrated.

[0048] Specifically, the amount of data to be migrated refers to the amount of data contained in the data stream to be migrated. This amount can be measured in data storage units such as GB or MB. The global data volume is the total amount of data contained in the source database; the occupancy rate is the ratio between the amount of data to be migrated and the global data volume; global memory information refers to the global memory space corresponding to the source database, used to temporarily store the migrated data after it has been migrated from the source database. Local memory information is the memory space allocated to the data stream to be migrated according to the ratio within the global memory information; local memory information represents the upper limit of memory that can be allocated to the data stream to be migrated.

[0049] Based on this, the amount of data to be migrated in the data stream to be migrated, and the global data volume corresponding to the source database, are determined. The occupancy rate of the data to be migrated within the global data volume of the source database is determined. The occupancy rate represents the proportion of the data to be migrated in the data stream within the global data volume, indicating the memory space requirement during data migration. Based on the occupancy rate and the global memory information of the source database, the local memory information corresponding to the data stream to be migrated is determined. A memory limit is set for the data stream to be migrated, allocating at most the memory space corresponding to the local memory information. The amount of data to be migrated serves as the initial migration data amount for the data stream, and the initial memory information for the initial memory space allocated when first allocating memory space for the data stream to be migrated can be determined based on this amount. The initial memory information and the local memory information are used as the memory allocation information for the data stream to be migrated.

[0050] For example, in a data migration scenario, full and incremental data from a source database can be migrated to a target database. The data to be migrated is read from the source database and temporarily stored in memory. Each source database corresponds to at least one data stream to be migrated, and the database migration for each data stream is performed synchronously; therefore, memory space needs to be allocated for each data stream. When allocating memory space, such as... Figure 3As shown, data streams 1, 2, 3, and 4 to be migrated are read from the source database. Memory constraints are applied to each data stream: initial memory space 1 is allocated for data stream 1, initial memory space 2 for data stream 2, initial memory space 3 for data stream 3, and initial memory space 4 for data stream 4. The initial memory space allocation can be based on the amount of data contained in each data stream.

[0051] In summary, by determining the memory allocation information of the data stream to be migrated based on the amount of data to be migrated and the global data volume of the source database, memory space can be allocated reasonably for the data stream to be migrated, thus avoiding the waste of storage resources.

[0052] Furthermore, since the source database can store a large amount of data, the amount of data in the data stream to be migrated, determined based on the source database, may also be substantial. In order to successfully migrate the data stream to the target database, it is necessary to allocate an appropriate initial memory space for the data stream to be migrated. The specific implementation is as follows:

[0053] The data volume of the data stream to be migrated is determined according to the memory allocation strategy, and the initial memory space of the data stream to be migrated is determined based on the data volume; or, the memory allocation information corresponding to the data stream to be migrated is determined according to the memory allocation strategy, and the initial memory space of the data stream to be migrated is determined based on the memory allocation information; or, the data volume of the data stream to be migrated and the memory allocation information corresponding to the data stream to be migrated are determined according to the memory allocation strategy, and the initial memory space of the data stream to be migrated is determined based on the data volume and the memory allocation information.

[0054] Specifically, the data volume of the data stream to be migrated represents its size, reflecting the storage space required during migration. Memory allocation information includes the memory space allocated to the data stream in a single migration and the overall memory allocation for the data stream. This memory allocation information refers to the upper limit of memory space that can be allocated to the data stream to be migrated. The memory allocation information can be determined based on the proportion of the data volume of the data stream to be migrated within the total global data volume to be migrated in the source database. The sum of the initial memory spaces of at least two data streams to be migrated from the source database must be less than the global memory space of the source database.

[0055] Based on this, when the memory allocation strategy corresponds to the data volume of the data stream to be migrated, an initial memory space matching the data volume of the data stream to be migrated can be allocated. When the memory allocation strategy corresponds to the memory allocation information of the data stream to be migrated, a fixed-size initial memory space can be allocated to the data stream to be migrated based on the memory allocation information of the data stream to be migrated. Correspondingly, when the memory allocation strategy corresponds to the data volume and memory allocation information of the data stream to be migrated, the proportion of the data volume corresponding to each data stream to be migrated in the total amount of data to be migrated in the source database is determined. This allows the memory allocation information to be updated to match the memory allocation information of each data stream to be migrated based on the proportion corresponding to each data stream to be migrated and the global memory space information of the source database. The memory allocation information includes a single memory space allocation, the initial memory space allocated to the data stream to be migrated, and the upper limit of the memory space corresponding to the data stream to be migrated.

[0056] Continuing with the previous example, if all four data streams to be migrated—1, 2, 3, and 4—correspond to the same data volume of 100MB, an initial memory space of 100MB matching the data volume of each stream can be allocated. Alternatively, an initial memory space matching the memory allocation information of each stream can be allocated. Based on the memory allocation information, a fixed-size memory space of 128MB can be pre-set for the source database, thus allocating 128MB of initial memory space to each stream in a single step. Furthermore, if the data volume of each stream to be migrated is determined to be 200MB, and the memory allocation information is 128MB, the memory allocation information can be updated from 128MB to 200MB, and an initial memory space of 200MB can be allocated.

[0057] In summary, the initial memory space can be determined based on the amount of data in the data stream to be migrated, the memory allocation information corresponding to the data stream to be migrated, or both the amount of data in the data stream to be migrated and the memory allocation information, thus enabling flexible determination of the initial memory space for the data stream to be migrated.

[0058] Step 204: During the data migration process, check whether the utilization rate of the initial memory space meets the memory usage conditions; if yes, proceed to step 206; if no, proceed to step 208.

[0059] Step 206: Release the memory space to be released in the initial memory space, and continue to migrate the data stream to be migrated using the memory space not released in the initial memory space;

[0060] Step 208: Expand the initial memory space into a target memory space according to the memory allocation information, and continue to migrate the data stream to be migrated using the target memory space.

[0061] Specifically, the utilization rate of the initial memory space can be calculated based on the memory information corresponding to the initial memory space and the memory usage information in the initial memory space. If the utilization rate of the initial memory space meets the memory usage conditions, it means that the initial memory space can store the data stream to be migrated. Conversely, if the utilization rate of the initial memory space does not meet the memory usage conditions, it means that the initial memory space is insufficient to store the data stream to be migrated. The memory to be released is the memory space in the initial memory space that has a set amount of memory and has not been used. The memory space not released refers to the memory space in the initial memory space that is still occupied by the data stream to be migrated. The target memory space refers to the memory space obtained after expanding the initial memory space.

[0062] Based on this, during the migration of the data stream to be migrated from the source database to the target database, the utilization rate of the initial memory space can be checked at fixed intervals to see if it meets the memory usage conditions. If the utilization rate of the initial memory space meets the memory usage conditions, it means that the initial memory space is sufficient to store the data stream to be migrated. At this time, it can be determined whether the initial memory space contains memory space to be released. If the initial memory space contains memory space to be released, the memory space to be released can be released, and the data migration of the data stream that has not been migrated can continue. If the utilization rate of the initial memory space does not meet the memory usage conditions, it means that incremental data to be migrated has been added to the data stream to be migrated, and the initial memory space cannot store the data to be migrated and the incremental data. At this time, it is necessary to expand the initial memory space to become the target memory space, so as to continue to migrate the data stream containing incremental data using the target memory space.

[0063] Furthermore, considering that the initial memory space is allocated for the data stream to be migrated, and that incremental data will be added during the migration process, the addition of incremental data may lead to insufficient initial memory space. In this case, it is necessary to determine whether more memory space can be allocated for the data stream to be migrated based on the control size of the initial memory space and the local memory information (memory space usage limit) in the memory allocation information. The specific implementation is as follows:

[0064] Based on the memory allocation information and the memory space information of the initial memory space, determine the memory space information to be allocated corresponding to the data stream to be migrated; if the memory space information to be allocated meets the memory acquisition conditions, determine the expansion memory space based on the memory allocation information and the memory space information to be allocated; expand the initial memory space into the target memory space based on the expansion memory space.

[0065] Specifically, the initial memory space information indicates the size of the initial memory space, such as 128MB; the unallocated memory space information is the amount of memory space that can still be allocated to the data stream to be migrated. Meeting the memory acquisition condition means that the amount of memory space that can still be allocated to the data stream to be migrated is greater than the initial memory space size. The memory acquisition condition is used to determine whether more memory space can be allocated to the data stream to be migrated.

[0066] Based on this, the memory space information to be allocated for the data stream to be migrated is determined according to the local memory information and the memory space information of the initial memory space in the memory allocation information. It is then checked whether the memory space information to be allocated meets the memory acquisition conditions. If the conditions are met, it means that the memory value corresponding to the memory space information to be allocated is greater than the memory value corresponding to the local memory information. In other words, the memory space usage of the data stream to be migrated has not reached the upper limit of memory space usage corresponding to the local memory information, and memory space can continue to be allocated for the data stream to be migrated, thus expanding the initial memory space. In other words, the expanded memory space is determined based on the memory allocation information and the memory space information to be allocated, and the initial memory space is expanded into the target memory space based on the expanded memory space.

[0067] Continuing with the previous example, if the maximum memory usage for the data stream to be migrated (1) is determined to be 1GB based on the local memory information in the memory allocation information, and considering the initial memory space of 100MB, the remaining 900MB of memory space can be allocated to the data stream to be migrated. Since this 900MB is greater than the 100MB that can be allocated to the data stream to be migrated in a single operation based on the initial memory information, this 100MB of memory space can be added to the initial memory space as an expansion.

[0068] In summary, based on memory allocation information and initial memory space information, it is determined whether to allocate expanded memory space for the data stream to be migrated. Expanded memory will not be allocated to the data stream to be migrated indefinitely, thus achieving reasonable and dynamic allocation of memory space.

[0069] Furthermore, considering that the memory space corresponding to the pending memory space information may be occupied by other data streams to be migrated, the occupancy status of the memory space corresponding to the pending memory space information is required when determining the memory space expansion. The specific implementation is as follows:

[0070] Determine whether the unoccupied memory space information in the pending memory space information meets the memory allocation conditions, wherein the unoccupied memory space information refers to the reserved memory space information in the pending memory space information that is not occupied by data streams other than the data stream to be migrated; if yes, execute the step of determining the expansion memory space based on the memory allocation information and the pending memory space information; if no, determine the target data stream in the source database, and determine the expansion memory space in the target memory space corresponding to the target data stream based on the memory allocation information.

[0071] Specifically, the unoccupied memory space information in the pending memory space information includes memory space that is not occupied by other data streams to be migrated. The determination of whether the unoccupied memory space information meets the memory allocation conditions involves checking whether the memory space corresponding to the unoccupied memory space information is greater than the memory space that can be allocated to the data stream to be migrated in a single allocation; in other words, whether the memory space corresponding to the unoccupied memory space information is sufficient to allocate to the data stream to be migrated. The target data stream can be any data stream in the source database that has memory space information that meets the memory allocation conditions.

[0072] Based on this, the unoccupied memory space information refers to the reserved memory space information in the pending memory space information that is not occupied by data streams other than the data stream to be migrated. It is determined whether the unoccupied memory space information in the pending memory space information meets the memory allocation conditions. If it does, it means that the unoccupied memory space corresponding to the unoccupied memory space information is sufficient to allocate to the data stream to be migrated. Therefore, the expansion memory space is determined based on the memory allocation information and the pending memory space information. If it does not meet the memory allocation conditions, it means that the unoccupied memory space corresponding to the unoccupied memory space information is insufficient to allocate to the data stream to be migrated. In this case, the target data stream is determined in the source database, and the expansion memory space is determined in the target memory space corresponding to the target data stream based on the memory allocation information. This achieves the "borrowing" of the pre-allocated memory space of the target data stream, solving the problem that the pre-allocated memory space of the data stream to be migrated is occupied, preventing the expansion of the initial memory space.

[0073] Continuing with the previous example, if the information on the unallocated memory space meets the memory acquisition conditions, it means that memory space can continue to be allocated for the data stream to be migrated. At this point, it is determined whether the information on unused memory space in the unallocated memory space information meets the memory allocation conditions, that is, whether the 90MB of unused memory space corresponding to the unused memory space information is greater than the 100MB of memory space allocated to data stream 1 in a single transaction. Clearly, the 90MB of unused memory space is not greater than the 100MB of memory space allocated to data stream 1 in a single transaction. Therefore, among data streams 2, 3, and 4 to be migrated, data stream 4 with a remaining unallocated memory space greater than 100MB can be identified. Data stream 4 is then selected as the target data stream, and 100MB of memory space is allocated from the unallocated memory space corresponding to data stream 4 as the expansion memory space.

[0074] In summary, when the unoccupied memory space information in the pending memory space information does not meet the memory allocation conditions, the expansion memory space is determined in the target memory space corresponding to the target data stream based on the memory allocation information. Thus, by "borrowing" the pre-allocated memory space of the target data stream, the problem of insufficient memory space required by the data stream to be migrated is solved.

[0075] Furthermore, considering that the initial memory space is used to temporarily store the data to be migrated from the source database, after the data to be migrated stored in the initial memory space is written to the target database, in order to improve the utilization of the initial memory space, the initial memory space can be released. The specific implementation is as follows:

[0076] Based on the target memory space usage information of the target memory space, determine the expanded memory space information, and determine the memory space occupancy information corresponding to the expanded memory space information; detect whether the memory space occupancy information is less than the initial memory reclamation threshold corresponding to the expanded memory space information;

[0077] If so, determine an initial release memory space in the target memory space that matches the initial memory reclamation threshold, and release the initial release memory space;

[0078] If not, determine whether the memory usage information corresponding to the target memory space is less than the target memory reclamation threshold corresponding to the data stream to be migrated; if it is less, continue to migrate the data stream to be migrated using the unreleased memory space in the target memory space; if it is greater, release the free memory space if the target memory space contains free memory space and a memory release instruction for the free memory space is received.

[0079] Specifically, the target memory space uses information to represent the numerical value of the target memory space; the expanded memory space information refers to the numerical value of the memory space expanded from the initial memory space in the target memory space; the memory space occupancy information corresponding to the expanded memory space information is the amount of memory space occupied by the expanded memory space in the target memory space; the initial memory reclamation threshold can be set to the numerical value of half the memory space of the expanded memory space in the target memory space; the target memory reclamation threshold corresponds to the local memory information in the memory allocation information, which is the upper limit of the memory space allocated for the data stream to be migrated. The memory release instruction is a computer instruction used to release the free memory space corresponding to the data stream to be migrated.

[0080] Based on this, the expanded memory space information is determined based on the target memory space usage information, and the corresponding memory space occupancy information is also determined. It is then checked whether the memory space occupancy information is less than the initial memory reclamation threshold corresponding to the expanded memory space information. If the memory space occupancy information is less than the initial memory reclamation threshold, it indicates that there is a significant amount of unused memory space in the target memory space, and an initial release memory space matching the initial memory reclamation threshold can be determined and released. If the memory space occupancy information is greater than or equal to the initial memory reclamation threshold, it indicates that there is a relatively small amount of unused memory space in the target memory space, and it can be determined whether the memory space usage information corresponding to the target memory space is less than the target memory reclamation threshold corresponding to the data stream to be migrated. If the memory usage information of the target memory space is less than the target memory reclamation threshold of the data stream to be migrated, it means that there is a small amount of unused memory space in the target memory space. If the memory space in the target memory space cannot be released, the unreleased memory space in the target memory space will be used to continue the data migration of the data stream to be migrated. If the memory usage information of the target memory space is greater than or equal to the target memory reclamation threshold of the data stream to be migrated, it means that there is a lot of unused memory space in the target memory space. If the target memory space contains free memory space and a memory release instruction for free memory space is received, the free memory space will be released in a targeted manner.

[0081] Continuing with the previous example, if the target memory space for the data stream to be migrated is 200MB, it means that an additional memory allocation has been made for data stream 1 to be migrated, based on the initial memory space. Therefore, 100MB is the expanded memory space. We then check if the memory usage of the 100MB expanded memory space is less than half of the expanded memory space (50MB). If the memory usage is 20MB, and the memory usage of the expanded memory space is less than half of the expanded memory space, then 50MB of memory space is allocated as the initial freed memory space. If the memory usage of the expanded memory space is 60MB, which is greater than or equal to half of the expanded memory space, we then check if the memory usage information of 160MB corresponding to the target memory space of 200MB is less than the upper limit of 1GB for the memory usage of data stream 1 to be migrated. If the memory usage information 160MB is less than the upper limit of memory usage 1GB corresponding to the data stream to be migrated, then the data migration of the data stream to be migrated continues. If the memory usage information is greater than or equal to the upper limit of memory usage 1GB corresponding to the data stream to be migrated, then the free memory space in the target memory space is released according to the memory release instruction.

[0082] In summary, based on the memory space occupancy information in the target memory space, memory is dynamically released from the target memory space, thereby flexibly shrinking the target memory space and improving memory space utilization.

[0083] Furthermore, considering that during the data migration process between the source and target databases, the data stream to be migrated is temporarily stored in the initial memory space, when there is a significant amount of free memory in the initial memory space, memory space can be released through memory shrinking. The specific implementation is as follows:

[0084] Based on a preset time interval, determine whether there is unoccupied memory space in the initial memory space that is greater than the memory space threshold; if so, use the unoccupied memory space as the memory space to be released, and execute the step of releasing the memory space to be released in the initial memory space; if not, continue to migrate the data stream to be migrated.

[0085] Specifically, the memory space threshold is used to detect whether unused memory space in the initial memory space needs to be released. If the value of unused memory space is greater than the memory space threshold, it indicates that there is a large amount of unused memory space in the initial memory space, and this unused memory space needs to be released. The preset time interval represents the detection period for the initial memory space and can be set according to requirements. The preset time interval is shorter than the migration period for data migration between the source and target databases.

[0086] Based on this, a preset time interval is used to determine whether there is unused memory space greater than the memory space threshold in the initial memory space. If there is unused memory space greater than the memory space threshold in the initial memory space, it indicates that there is a large amount of unused memory space, which needs to be released. This unused memory space can be designated as memory space to be released, and the unused memory space in the initial memory space can be released accordingly. Alternatively, memory space matching the memory space threshold can be identified from the unused memory space and released accordingly. If there is unused memory space greater than the memory space threshold in the initial memory space, it indicates that there is a small amount of unused memory space, which does not need to be released, and data migration can continue. Alternatively, a preset time interval is used to determine whether there is unused memory space greater than the memory space threshold in the initial memory space, and the unused time exceeds the time threshold. When there is unused memory space greater than the memory space threshold in the initial memory space and the unused time reaches the time threshold, this unused memory space is released as memory space to be released.

[0087] Using the previous example, if the memory space threshold is set to 50MB, and if 60MB of memory space is found to be unoccupied within the initial 100MB memory space and the unoccupied time is greater than 30 seconds, then 50MB of memory space will be selected from the 60MB of unoccupied memory space to be released.

[0088] In summary, by determining whether there is unused memory space greater than the memory space threshold in the initial memory space, it is possible to detect in a timely manner whether there is a large amount of unused memory space in the initial memory space, and then release the unused memory space for use when other data streams to be migrated, thereby improving the utilization rate of memory space.

[0089] Furthermore, considering that the memory space corresponding to the data stream to be migrated is only used to temporarily store the data stream to be migrated read from the source database, the memory space can be released after the data stream to be migrated in the memory space is written to the target database. During the migration of the data stream to be migrated, the memory space that is not occupied can also be released. The specific implementation is as follows:

[0090] If the data migration of the data stream to be migrated is carried out using the unreleased memory space in the initial memory space and the migration is completed, the unreleased memory space is released; or, if the data migration of the data stream to be migrated is carried out using the target memory space and the migration is completed, the target memory space is released.

[0091] Therefore, the initial memory space is used to temporarily store the data stream to be migrated during the migration process. Once the migration of the data stream is completed using the unreleased memory space in the initial memory space, it indicates that the data corresponding to the data stream to be migrated no longer exists in the initial memory space, and the initial memory space is no longer occupied. The unreleased memory space can then be released, completing the release of the initial memory space. Conversely, once the migration of the data stream to be migrated continues using the target memory space, and the migration is completed, it indicates that the migration process of the data stream to be migrated is finished, the target memory space will no longer be used, and the target memory space can then be released.

[0092] In summary, during the migration of the data stream, once the migration is complete, the initial or target memory space occupied by the data stream can be released, thus timely reclaiming unused memory space. The reclaimed memory space can then be used for the transmission of other data, thereby improving the utilization rate of memory resources.

[0093] In one embodiment of this specification, during the migration of data to be migrated from the source database to the target database, the data to be migrated is abstracted into a data stream and migrated to the target database. Based on the initial memory space usage rate of the data stream during the data migration process, the size of the initial memory space is dynamically adjusted. That is, when the initial memory space is sufficient, unused memory space in the initial memory space is released; when the initial memory space is insufficient, the initial memory space is expanded. This achieves dynamic adjustment of the memory space according to the memory space usage requirements during the data migration process, thereby fully and rationally utilizing memory space, effectively increasing migration concurrency, and improving data migration efficiency.

[0094] The following is in conjunction with the appendix Figure 4 Taking the application of the data migration method provided in this specification in data migration between databases as an example, the data migration method will be further explained. Figure 4 This specification illustrates a flowchart of a data migration method according to an embodiment, which specifically includes the following steps:

[0095] Step 402: Determine the source database and the target database.

[0096] Step 404: Identify the data table to be migrated in the source database and determine the data flow corresponding to the data table to be migrated.

[0097] In practice, the source database contains at least one table to be migrated. If the source database contains tables A, B, C, and D to be migrated, each table corresponds to a data stream. That is, table A corresponds to data stream 1, table B to data stream 2, table C to data stream 3, and table D to data stream 4. The four tables can be migrated concurrently.

[0098] Step 406: Allocate initial memory space for the data stream corresponding to the data table to be migrated.

[0099] Allocate 128MB of initial memory to the data stream corresponding to each data table to be migrated.

[0100] Step 408: Migrate the data to be migrated from the data table to the initial memory space for temporary storage using the data stream.

[0101] Step 410: Migrate the data to be migrated temporarily stored in the initial memory space to the target database.

[0102] Step 412: During the data migration process, determine whether the initial memory space meets the memory usage conditions. If yes, proceed to step 414; otherwise, proceed to step 416.

[0103] During data migration, memory allocation and reclamation (release) can be dynamically implemented. Determining whether the initial memory space meets the memory usage conditions is actually determining whether the initial memory space can support the migration of the data to be migrated, or whether the initial memory space can meet the memory usage requirements.

[0104] Step 414: Release the memory space to be released in the initial memory space, and continue to migrate the data stream to be migrated using the memory space not released in the initial memory space.

[0105] If the initial memory space meets the memory usage requirements, it means that the initial memory space is sufficient for the data in the data table to be migrated to complete the data migration. At this point, once the data migration in the data table is complete, the initial memory space can be released. Alternatively, if there is free memory in the initial memory space that will not be used for a long time, it can be released as memory space to be freed up.

[0106] Step 416: Expand the initial memory space into the target memory space according to the memory allocation information, and continue to migrate the data stream to be migrated using the target memory space.

[0107] If the initial memory space does not meet the memory usage requirements, it indicates that the initial memory space is insufficient. Memory expansion is needed. At this point, an additional 128MB of memory can be requested to support data migration. If the memory allocation is successful, data migration continues. If the memory allocation fails, and the already allocated memory does not exceed the average allocated memory, memory can be obtained from other migration jobs; if the memory allocation fails, and the already allocated memory exceeds the average allocated memory, memory limiting is applied. This allows for dynamic memory expansion and release based on memory usage, effectively increasing incremental migration concurrency and accelerating the incremental migration rate. It also improves memory resource utilization.

[0108] In practical applications, memory usage is limited when migrating data between a source database and a target database. For example... Figure 5 As shown, a data migration task is performed on the source database. Data tables are read from the source database, and it is determined whether memory usage exceeds the limit. If memory usage exceeds the limit, the process waits for memory to be released; otherwise, memory usage is counted. Data to be migrated from the source database is temporarily stored in memory. A migration task is then performed, writing the temporarily stored data from memory to the target database and decrementing the memory count, effectively releasing the memory.

[0109] The execution logic for the migration-out and migration-in tasks is as follows: Figure 6 As shown. During the migration task, after allocating initial memory to the data table to be migrated in the source database, it is determined whether it is necessary to continue allocating memory to the data table to be migrated. If not, the data migration can continue. If it is necessary, it is determined whether it is possible to continue allocating memory to the data table to be migrated. If so, an additional 128MB of memory is allocated to the data table to be migrated. If not, it is determined whether the memory currently allocated to the data table to be migrated is greater than the average allocated memory. If it is greater, the data table to be migrated is rate-limited. If it is less, the memory of other data streams can be reduced.

[0110] During the migration process, it is determined whether the data table to be migrated has completed the migration. If the migration is complete, the memory allocated to the data table is released. If the migration is incomplete, it is determined whether the memory usage is less than half of the allocated memory. If so, the memory allocated to the data table is released. If not, it is determined whether the memory usage is less than the average allocated memory. If so, the data migration continues. If not, it is determined whether the memory usage exceeds the limit. If the memory usage exceeds the limit, the memory is released; otherwise, the data migration continues. During the process of migrating data from the source database to memory, memory allocation is flexibly performed. Correspondingly, during the process of writing data temporarily stored in memory to the target database, memory that is no longer in use is released. Memory is dynamically expanded and shrunk, thereby improving the utilization rate of memory resources and achieving flexible management of memory resources.

[0111] During the migration of data from the source database to the target database, the data is abstracted into a data stream and migrated to the target database. Based on the initial memory usage of the data stream during the migration process, the size of the initial memory space is dynamically adjusted. Specifically, if the initial memory space is sufficient, unused memory is released; if the initial memory space is insufficient, it is expanded. This dynamic adjustment of memory space according to the memory usage requirements during data migration ensures full and rational utilization of memory, effectively increases migration concurrency, and improves data migration efficiency.

[0112] Corresponding to the above method embodiments, this specification also provides embodiments of a data migration apparatus. Figure 7 A schematic diagram of a data migration apparatus according to one embodiment of this specification is shown. Figure 7 As shown, the device includes:

[0113] Migration module 702 is configured to determine the data stream to be migrated in the source database and migrate the data stream to be migrated from the source database to the target database through an initial memory space, wherein the initial memory space is determined according to a memory allocation strategy;

[0114] The judgment module 704 is configured to detect whether the utilization rate of the initial memory space meets the memory usage conditions during the data migration process;

[0115] If so, the release module 706 is run. The release module 706 is configured to release the memory space to be released in the initial memory space and continue to migrate the data stream to be migrated using the unreleased memory space in the initial memory space.

[0116] If not, the expansion module 708 is run. The expansion module 708 is configured to expand the initial memory space into a target memory space according to the memory allocation information, and use the target memory space to continue to migrate the data stream to be migrated.

[0117] In an optional embodiment, the determining module 704 is further configured to:

[0118] The amount of data to be migrated in the data stream to be migrated is determined, and the occupancy rate of the data to be migrated in the global data volume corresponding to the source database is determined; based on the occupancy rate and the global memory information corresponding to the source database, the local memory information corresponding to the data stream to be migrated is determined; the initial memory information is determined according to the amount of data to be migrated, and the initial memory information and the local memory information are used as the memory allocation information corresponding to the data stream to be migrated.

[0119] In an optional embodiment, the release module 706 is further configured to:

[0120] Based on a preset time interval, determine whether there is unoccupied memory space in the initial memory space that is greater than the memory space threshold; if so, treat the unoccupied memory space as memory space to be released and execute the step of releasing the memory space to be released in the initial memory space; if not, continue to migrate the data stream to be migrated.

[0121] In an optional embodiment, the expansion module 708 is further configured to:

[0122] Based on the memory allocation information and the memory space information of the initial memory space, determine the memory space information to be allocated corresponding to the data stream to be migrated; if the memory space information to be allocated meets the memory acquisition conditions, determine the expansion memory space based on the memory allocation information and the memory space information to be allocated; expand the initial memory space into the target memory space based on the expansion memory space.

[0123] In an optional embodiment, the expansion module 708 is further configured to:

[0124] The memory space information to be allocated corresponding to the data stream to be migrated is determined based on the memory allocation information and the memory space information of the initial memory space; if the memory space information to be allocated meets the memory acquisition conditions, the memory space to be expanded is determined based on the memory allocation information and the memory space information to be allocated; the initial memory space is expanded into the target memory space based on the expanded memory space.

[0125] In an optional embodiment, the expansion module 708 is further configured to:

[0126] Determine whether the unoccupied memory space information in the pending memory space information meets the memory allocation conditions. The unoccupied memory space information refers to the reserved memory space information in the pending memory space information that is not occupied by data streams other than the data stream to be migrated. If yes, execute the step of determining the expansion memory space based on the memory allocation information and the pending memory space information. If no, determine the target data stream in the source database, and determine the expansion memory space in the target memory space corresponding to the target data stream based on the memory allocation information.

[0127] In an optional embodiment, the expansion module 708 is further configured to:

[0128] Based on the target memory space usage information, determine the expanded memory space information and the corresponding memory space occupancy information; check if the memory space occupancy information is less than the initial memory reclamation threshold corresponding to the expanded memory space information; if so, determine the initial release memory space in the target memory space that matches the initial memory reclamation threshold and release the initial release memory space; if not, determine if the memory space usage information corresponding to the target memory space is less than the target reclamation memory threshold corresponding to the data stream to be migrated; if less, continue data migration of the data stream to be migrated using the unreleased memory space in the target memory space; if greater, release the free memory space if the target memory space contains free memory space and a memory release instruction for the free memory space is received.

[0129] In an optional embodiment, the expansion module 708 is further configured to:

[0130] If the unreleased memory space in the initial memory space is used to migrate the data stream to be migrated, and the migration is completed, the unreleased memory space is released; or, if the data stream to be migrated is continued to be migrated using the target memory space, and the migration is completed, the target memory space is released.

[0131] In an optional embodiment, the migration module 702 is further configured to:

[0132] The amount of data to be migrated in the data stream to be migrated is determined, and the occupancy rate of the data to be migrated in the global data volume corresponding to the source database is determined; based on the occupancy rate and the global memory information corresponding to the source database, the local memory information corresponding to the data stream to be migrated is determined; the initial memory information is determined according to the amount of data to be migrated, and the initial memory information and the local memory information are used as the memory allocation information corresponding to the data stream to be migrated.

[0133] In an optional embodiment, the migration module 702 is further configured to:

[0134] The data volume of the data stream to be migrated is determined according to the memory allocation strategy, and the initial memory space of the data stream to be migrated is determined based on the data volume; or, the memory allocation information corresponding to the data stream to be migrated is determined according to the memory allocation strategy, and the initial memory space of the data stream to be migrated is determined based on the memory allocation information; or, the data volume of the data stream to be migrated and the memory allocation information corresponding to the data stream to be migrated are determined according to the memory allocation strategy, and the initial memory space of the data stream to be migrated is determined based on the data volume and the memory allocation information.

[0135] In summary, during the migration of data from the source database to the target database, the data is abstracted into a data stream and migrated to the target database. Based on the initial memory usage of the data stream during the migration process, the size of the initial memory space is dynamically adjusted. Specifically, if the initial memory space is sufficient, unused memory is released; if the initial memory space is insufficient, it is expanded. This dynamic adjustment of memory space according to the memory usage requirements during data migration ensures full and rational utilization of memory, effectively increases migration concurrency, and improves data migration efficiency.

[0136] The above is an illustrative scheme of a data migration device according to this embodiment. It should be noted that the technical solution of this data migration device and the technical solution of the data migration method described above belong to the same concept. For details not described in detail in the technical solution of the data migration device, please refer to the description of the technical solution of the data migration method described above.

[0137] Corresponding to the above method embodiments, this specification also provides data migration system embodiments. Figure 8 A schematic diagram of the structure of a data migration system provided in one embodiment of this specification is shown. Figure 8 As shown, the data processing system 800 includes a source data processing node 810 corresponding to the source database 812, a memory management node 820, and a target data processing node 830 corresponding to the target database 832.

[0138] The source data processing node 810 is used to determine the data stream to be migrated in the source database 812 and migrate the data stream to be migrated to the initial memory space, wherein the initial memory space is determined according to the memory allocation strategy;

[0139] The memory management node 820 is used to detect whether the utilization rate of the initial memory space meets the memory usage conditions during the data migration process; if so, it releases the memory space to be released in the initial memory space and continues to migrate the data stream to be migrated using the unreleased memory space in the initial memory space; if not, it expands the initial memory space into a target memory space according to the memory allocation information and continues to migrate the data stream to be migrated using the target memory space.

[0140] The target data processing node 830 is used to send feedback information indicating that the data migration is complete to the source data processing node 810 when the data stream to be migrated is completed.

[0141] In practical applications, the data processing system 800 includes a source data processing node 810 corresponding to the source database 812, a memory management node 820, and a target data processing node 830 corresponding to the target database 832. The memory management node 820 is used to allocate the required memory space for the data stream to be migrated during the migration process from the source database to the target database. It also releases memory space no longer occupied by the data stream during the migration process and releases the memory space occupied by the data stream after migration to the target database. The memory management node 820 is used for allocating and reclaiming memory space. After determining the data stream to be migrated in the source database, the memory management node 820 allocates initial memory space for the data stream. The source database processing node migrates the data stream to the initial memory space for temporary storage. The target data processing node 830 reads the data stream from the initial memory space and stores it in the target database. During the process of storing the data stream to be migrated into the initial memory space, incremental data corresponding to the data stream to be migrated may be generated in the source database. Therefore, the incremental data also needs to be added to the data stream to be migrated. At this time, the initial memory space may be insufficient. The initial memory space can be expanded to become the target memory space. Correspondingly, during the process of writing the data to be migrated from the initial memory space to the target database, as the data stream to be migrated from the initial memory space is moved out and written to the target database, the initial memory space will generate free memory space. This free memory space can be used as memory space to be released, and the memory management node 820 will release the memory space to be released.

[0142] In summary, during the migration of data from the source database to the target database, the data is abstracted into a data stream and migrated to the target database. Based on the initial memory usage of the data stream during the migration process, the size of the initial memory space is dynamically adjusted. Specifically, if the initial memory space is sufficient, unused memory is released; if the initial memory space is insufficient, it is expanded. This dynamic adjustment of memory space according to the memory usage requirements during data migration ensures full and rational utilization of memory, effectively increases migration concurrency, and improves data migration efficiency.

[0143] The above is an illustrative scheme of a data migration system according to this embodiment. It should be noted that the technical solution of this data migration system and the technical solution of the data migration method described above belong to the same concept. For details not described in detail in the technical solution of the data migration system, please refer to the description of the technical solution of the data migration method described above.

[0144] Figure 9 A structural block diagram of a computing device 900 according to one embodiment of this specification is shown. The components of the computing device 900 include, but are not limited to, a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0145] The computing device 900 also includes an access device 940, which enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 940 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0146] In one embodiment of this specification, the above-described components of the computing device 900 and Figure 9 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 9 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0147] The computing device 900 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 900 can also be a mobile or stationary server.

[0148] The processor 920 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-described data migration method.

[0149] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data migration method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data migration method described above.

[0150] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data migration method described above.

[0151] The above is an illustrative embodiment of a computer-readable storage medium. It should be noted that the technical solution of this storage medium and the technical solution of the data migration method described above belong to the same concept. Details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the data migration method described above.

[0152] An embodiment of this specification also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described data migration method.

[0153] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the data migration method described above belong to the same concept. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solution of the data migration method described above.

[0154] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0155] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0156] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0157] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0158] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data migration method, comprising: The data stream to be migrated is determined in the source database, and the data stream to be migrated is migrated from the source database to the target database through an initial memory space, wherein the initial memory space is determined according to a memory allocation strategy; During the data migration process, the utilization rate of the initial memory space is checked to see if it meets the memory usage conditions. If so, the memory space to be released in the initial memory space is released, and the unreleased memory space in the initial memory space is used to continue the data migration of the data stream to be migrated; If not, the initial memory space is expanded into a target memory space according to the memory allocation information, and the target memory space is used to continue migrating the data stream to be migrated.

2. The data migration method according to claim 1, wherein expanding the initial memory space into a target memory space according to the memory allocation information comprises: The memory space information to be allocated corresponding to the data stream to be migrated is determined based on the memory allocation information and the memory space information of the initial memory space. If the information of the memory space to be allocated meets the memory acquisition conditions, the memory space to be expanded is determined according to the memory allocation information and the information of the memory space to be allocated; The initial memory space is expanded into the target memory space based on the expanded memory space.

3. The data migration method according to claim 2, wherein determining the expanded memory space based on the memory allocation information and the memory space to be allocated includes: Determine whether the unoccupied memory space information in the pending memory space information meets the memory allocation conditions, wherein the unoccupied memory space information refers to the reserved memory space information in the pending memory space information that is not occupied by data streams other than the data stream to be migrated; If so, execute the step of determining the expanded memory space based on the memory allocation information and the memory space to be allocated; If not, determine the target data stream in the source database, and determine the expansion memory space in the target memory space corresponding to the target data stream according to the memory allocation information.

4. The data migration method according to claim 1, further comprising: Based on the target memory space usage information of the target memory space, the expanded memory space information is determined, and the memory space occupancy information corresponding to the expanded memory space information is determined; Detect whether the memory space usage information is less than the initial memory reclamation threshold corresponding to the expanded memory space information; If so, determine an initial release memory space in the target memory space that matches the initial memory reclamation threshold, and release the initial release memory space; If not, determine whether the memory usage information corresponding to the target memory space is less than the target memory reclamation threshold corresponding to the data stream to be migrated; If the value is less than the target memory space, the unreleased memory space in the target memory space will be used to continue the data migration of the data stream to be migrated. If the value is greater than the target memory space, and a memory release instruction for the free memory space is received, the free memory space is released.

5. The data migration method according to claim 1, further comprising, before releasing the memory space to be released in the initial memory space: Based on a preset time interval, determine whether there is unoccupied memory space in the initial memory space that is greater than the memory space threshold; If so, the unoccupied memory space is used as the memory space to be released, and the step of releasing the memory space to be released in the initial memory space is executed; If not, continue migrating the data stream to be migrated.

6. The data migration method according to claim 1, further comprising: After migrating the data stream to be migrated using the unreleased memory space in the initial memory space, and after the migration is completed, the unreleased memory space is released. or, After the data migration of the data stream to be migrated is continued using the target memory space and the migration is completed, the target memory space is released.

7. The data migration method according to any one of claims 1-3, wherein determining the memory allocation information corresponding to the data stream to be migrated includes: Determine the amount of data to be migrated in the data stream to be migrated, and determine the percentage of the amount of data to be migrated in the global data volume corresponding to the source database; Based on the occupancy rate and the global memory information corresponding to the source database, determine the local memory information corresponding to the data stream to be migrated; The initial memory information is determined based on the amount of data to be migrated, and the initial memory information and the local memory information are used as the memory allocation information corresponding to the data stream to be migrated.

8. The data migration method according to claim 1, wherein determining the initial memory space according to the memory allocation strategy comprises: The amount of data in the data stream to be migrated is determined according to the memory allocation strategy, and the initial memory space of the data stream to be migrated is determined based on the amount of data. or, The memory allocation information corresponding to the data stream to be migrated is determined according to the memory allocation strategy, and the initial memory space of the data stream to be migrated is determined based on the memory allocation information. or, The data volume of the data stream to be migrated and the memory allocation information corresponding to the data stream to be migrated are determined according to the memory allocation strategy, and the initial memory space of the data stream to be migrated is determined based on the data volume and the memory allocation information.

9. A data migration system, comprising a memory management node, a source data processing node corresponding to the source database, and a target data processing node corresponding to the target database; The source data processing node is used to determine the data stream to be migrated in the source database and migrate the data stream to be migrated to the initial memory space, wherein the initial memory space is determined according to a memory allocation strategy; The memory management node is used to detect whether the utilization rate of the initial memory space meets the memory usage conditions during the data migration process; if so, it releases the memory space to be released in the initial memory space and continues to migrate the data stream to be migrated using the unreleased memory space in the initial memory space; if not, it expands the initial memory space into a target memory space according to the memory allocation information and continues to migrate the data stream to be migrated using the target memory space. The target data processing node is used to send feedback information indicating that the data migration is complete to the source data processing node when the data stream to be migrated is completed.

10. A data migration apparatus, comprising: The migration module is configured to determine the data stream to be migrated in the source database and migrate the data stream to be migrated from the source database to the target database through an initial memory space, wherein the initial memory space is determined according to a memory allocation strategy; The judgment module is configured to detect whether the utilization rate of the initial memory space meets the memory usage conditions during the data migration process; If so, the release module is run. The release module is configured to release the memory space to be released in the initial memory space, and to continue to migrate the data stream to be migrated using the unreleased memory space in the initial memory space. If not, the expansion module is run. The expansion module is configured to expand the initial memory space into a target memory space according to the memory allocation information, and use the target memory space to continue to migrate the data stream to be migrated.

11. A computing device, comprising: Memory and processor; The memory is used to store computer programs or instructions, and the processor is used to execute the computer programs or instructions, which, when executed by the processor, implement the steps of the data migration method according to any one of claims 1 to 8.

12. A computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement the steps of the data migration method according to any one of claims 1 to 8.

13. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the steps of the data migration method of any one of claims 1 to 8.

Images