Data migration method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data migration method and device, electronic equipment and storage medium, which can be applied to the fields of computer technology, big data, finance or other fields. The method comprises: in response to a data migration request, determining a first target field from a to-be-migrated data table recorded in a source database; determining the number of bytes occupied by the first target field based on the data stored in the first target field; updating a data migration configuration parameter based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the to-be-migrated data table; and executing a data migration task included in the data migration request to migrate the to-be-migrated data recorded in the source database to a target database by using a data migration tool, wherein the to-be-migrated data comprises data stored in the to-be-migrated data table.

Description

Technical Field

[0001] This disclosure relates to the fields of computer technology, big data, and finance, and more specifically, to a data migration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Data migration can move data from one database to another, making it easier for users to update systems or replace equipment. Therefore, with the rapid development of information systems, data migration is particularly important.

[0003] There are many existing data migration tools, but they still have certain shortcomings. For example, when using Data Replication Service (DRS) for data migration, if migrating from an Oracle database to a Gauss database, and if the Oracle table contains large objects, a memory leak may occur during the migration, causing the migration task to be interrupted. Summary of the Invention

[0004] In view of the above, this disclosure provides a data migration method, apparatus, electronic device, readable storage medium, and computer program product.

[0005] One aspect of this disclosure provides a data migration method, comprising:

[0006] In response to a data migration request, the first target field is determined from the data table to be migrated recorded in the source database;

[0007] Based on the data stored in the first target field, determine the number of bytes occupied by the first target field.

[0008] Based on the number of bytes occupied by the first target field and its relationship with the data table to be migrated, update the data migration configuration parameters; and

[0009] The data migration request includes a data migration task to use a data migration tool to migrate the data to be migrated from the source database to the target database, wherein the data to be migrated includes the data stored in the data table to be migrated.

[0010] According to an embodiment of this disclosure, the above-mentioned method of determining a first target field from a data table to be migrated recorded in the source database in response to a data migration request includes: in response to the data migration request, using a system view of the source database, determining the field type of each of the plurality of data fields recorded in the data table to be migrated; and determining the first target field that satisfies preset conditions from the plurality of data fields, wherein the preset conditions include the field type of the data field being a preset field type.

[0011] According to embodiments of this disclosure, updating the data migration configuration parameters based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the data table to be migrated includes: determining first byte count information based on the number of bytes occupied by the first target field; determining second byte count information based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the data table to be migrated; determining third byte count information based on the second byte count information; and updating the data migration configuration parameters using the first byte count information, the second byte count information, and the third byte count information respectively.

[0012] According to embodiments of this disclosure, the data migration configuration parameters include process configuration parameters, environment configuration parameters, and task configuration parameters. The method of updating the data migration configuration parameters using the first byte information, the second byte information, and the third byte information includes: using a first preset command to log in to the business node associated with the data migration tool based on the user authentication information included in the data migration request; using the first byte information to update the process configuration parameters of the business node; using a second preset command to update the environment configuration parameters of the virtual machine associated with the business node based on the second byte information; and using the third byte information to update the task configuration parameters of the data migration task.

[0013] According to an embodiment of this disclosure, determining the first byte count information based on the number of bytes occupied by the first target field includes: determining a second target field from the first target field based on the number of bytes occupied by the first target field; and determining the first byte count information based on the number of bytes occupied by the second target field.

[0014] According to embodiments of this disclosure, determining the second byte count information based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the data table to be migrated includes: dividing the first target field into at least one field set based on the dependency relationship between the first target field and the data table to be migrated, wherein the field set corresponds to the data table to be migrated; determining a third target field from the first target fields included in each field set; and determining the second byte count information based on the number of bytes occupied by the third target field in each of the at least one field set.

[0015] According to embodiments of this disclosure, determining the third byte number information based on the second byte number information includes: determining the fourth byte number information based on the fixed parameters of the data migration tool and the second byte number information; and rounding the fourth byte number information to obtain the third byte number information.

[0016] Another aspect of this disclosure provides a data migration apparatus, comprising:

[0017] The first target field determination module is used to determine the first target field from the data table to be migrated recorded in the source database in response to a data migration request.

[0018] The byte count determination module is used to determine the number of bytes occupied by the first target field based on the data stored in the first target field.

[0019] The configuration parameter update module is used to update the data migration configuration parameters based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the data table to be migrated; and

[0020] The data migration module is used to execute the data migration tasks included in the data migration request, so as to use the data migration tool to migrate the data to be migrated recorded in the source database to the target database, wherein the data to be migrated includes the data stored in the data table to be migrated.

[0021] Another aspect of this disclosure provides an electronic device, including: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0022] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0023] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the method described above.

[0024] According to embodiments of this disclosure, before data migration, it can be detected whether the source database contains a first target field. If the target field is found to be present, the data migration configuration parameters are updated based on the number of bytes occupied by each first target field to be migrated. After the parameter configuration is completed, data migration from the source database to the target database is performed. The first target field can be a large object field. By using the storage parameters of the large object field to configure the data migration configuration parameters, such as memory size, the amount of data migrated in a single data migration task can be less than the amount of data limited by the data migration configuration parameters. Therefore, this at least partially overcomes the memory overflow problem caused by the presence of large object fields in the migrated data during the data migration process, better ensuring the security and integrity of the data migration process. Attached Figure Description

[0025] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0026] Figure 1 This illustration schematically shows an exemplary system architecture to which data migration methods and apparatus can be applied according to embodiments of this disclosure;

[0027] Figure 2 A flowchart illustrating a data migration method according to an embodiment of the present disclosure is shown schematically.

[0028] Figure 3 A flowchart illustrating the determination of a first target field according to an embodiment of the present disclosure is shown schematically;

[0029] Figure 4 A flowchart illustrating the data migration configuration parameter update process according to an embodiment of the present disclosure is shown schematically.

[0030] Figure 5 A block diagram of a data migration apparatus according to embodiments of the present disclosure is schematically shown; and

[0031] Figure 6 A block diagram of an electronic device suitable for implementing a data migration method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0032] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0034] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0035] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0036] During the research, it was found that data migration often involves the problem of data being interrupted due to memory leaks during the migration process. For example, when migrating from Oracle to Gauss database, if the Oracle table contains large objects, a memory leak may occur during the migration, thus causing the migration task to be interrupted.

[0037] In view of the above, embodiments of this disclosure provide a data migration method, a data migration apparatus, an electronic device, a readable storage medium, and a computer program product. The method includes, in response to a data migration request, determining a first target field from a data table to be migrated recorded in a source database; determining the number of bytes occupied by the first target field based on the data stored in the first target field; updating data migration configuration parameters based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the data table to be migrated; and executing a data migration task included in the data migration request to migrate the data to be migrated recorded in the source database to the target database using a data migration tool, wherein the data to be migrated includes the data stored in the data table to be migrated.

[0038] It should be noted that the data migration method and apparatus determined in the embodiments of this disclosure can be used in the fields of computer technology, big data, or finance. The data migration method and apparatus determined in the embodiments of this disclosure can also be used in any field other than computer technology and finance; therefore, the application fields of the data migration method and apparatus determined in the embodiments of this disclosure are not limited.

[0039] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0040] Figure 1 This illustration schematically depicts an exemplary system architecture to which data migration methods and apparatus can be applied according to embodiments of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0041] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first database 101, terminal devices 102, 103, and 104, and a second database 105. The first database 101 and terminal devices 102, 103, and 104 can be connected via a network, and the second database 105 can also be connected via a network. The network serves as a medium for providing communication links between terminal devices 102, 103, and 104 and the first and second databases 101 and 105. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0042] Terminal devices 102, 103, and 104 may be equipped with DRS tools or other data migration tools, which interact with the first database 101 and the second database 105 via network 104 to receive or send data. Various communication client applications may be installed on terminal devices 102, 103, and 104, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platforms (examples only).

[0043] Terminal devices 102, 103, and 104 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0044] The first database 101 can be a source database storing data to be migrated; the second database 105 can be a target database receiving the migrated data.

[0045] It should be noted that the data migration method provided in this embodiment can generally be executed by terminal devices 102, 103, and 104, or by other terminal devices different from terminal devices 102, 103, or 104. Accordingly, the data migration apparatus provided in this embodiment can generally be disposed in terminal devices 102, 103, and 104, or in other terminal devices different from terminal devices 104, 102, or 103.

[0046] For example, the DRS data migration tool can be installed in any one of terminal devices 102, 103, or 104 (e.g., terminal devices, but not limited thereto). The terminal device can then execute the data migration method provided in the embodiments of this disclosure locally.

[0047] It should be understood that Figure 1 The number of terminal devices and databases shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers.

[0048] Figure 2 A flowchart illustrating a data migration method according to an embodiment of the present disclosure is shown schematically.

[0049] like Figure 2 As shown, the method includes operations S201 to S204.

[0050] In operation S201, in response to a data migration request, the first target field is determined from the data table to be migrated recorded in the source database.

[0051] According to embodiments of this disclosure, a first target field is obtained by querying a data table to be migrated recorded in the source database using a predetermined method based on a data migration request.

[0052] According to embodiments of this disclosure, the type of the source database is not limited and can be an Oracle database.

[0053] According to embodiments of this disclosure, the data table to be migrated typically includes multiple fields, among which the field with the largest content can be a large object field, that is, the first target field can be a large object field.

[0054] According to embodiments of this disclosure, by statistically analyzing all first target fields in the source database, information about large object fields can be better grasped, fields that may cause memory leaks can be identified, and memory leaks can be avoided as much as possible based on this.

[0055] In operation S202, the number of bytes occupied by the first target field is determined based on the data stored in the first target field.

[0056] According to embodiments of this disclosure, after calculating the number of bytes occupied by the first target field in each data table to be migrated, the number of bytes of all first target fields in each data table to be migrated is recorded to obtain a byte count statistics table for each data table to be migrated. The specific method for calculating the number of bytes occupied by the first target field is not limited; it can be calculated using the dbms_lob.get_length function.

[0057] According to the embodiments of this disclosure, by counting the number of bytes occupied by the first target field of each data table to be migrated, the number of bytes occupied by the first target field of each data table to be migrated can be obtained more clearly and intuitively. Based on this number of bytes, subsequent settings can be made to avoid the problem of memory overflow that may occur during data migration due to insufficient allocated memory to fully store the migrated data.

[0058] In operation S203, the data migration configuration parameters are updated based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the data table to be migrated.

[0059] According to embodiments of this disclosure, the maximum number of bytes in each data table to be migrated is determined from the number of bytes occupied by the first target field in the data table to be migrated, wherein the maximum number of bytes in each data table to be migrated refers to the number of bytes occupied by the first target field with the largest number of bytes in each data table to be migrated.

[0060] According to embodiments of this disclosure, data migration configuration parameters are updated using the maximum number of bytes in each data table to be migrated and the dependency relationship between the first target field occupying the maximum number of bytes and the data table to be migrated.

[0061] According to embodiments of this disclosure, the number of bytes occupied by the first target field and the subordinate relationship between the first target field and the data table to be migrated are used to update the data migration configuration parameters, so that the data migration configuration parameters can completely migrate the data to be migrated in each data migration data table to the target database. This at least partially overcomes the problem that the target database memory may not be able to store all the data in each data table to be transmitted when transmitting directly, thereby ensuring the integrity of the data.

[0062] In operation S204, the data migration request includes a data migration task to use a data migration tool to migrate the data to be migrated from the source database to the target database, wherein the data to be migrated includes the data stored in the data table to be migrated.

[0063] According to embodiments of this disclosure, a data migration tool can be used to execute the data migration task included in the data migration request, and finally complete the data migration from the source database to the target database.

[0064] According to embodiments of this disclosure, the target database can be a Gauss database.

[0065] According to embodiments of this disclosure, the data migration tool can be a DRS data migration tool.

[0066] According to embodiments of this disclosure, before data migration, it can be detected whether the source database contains a first target field. If the target field is found to be present, the data migration configuration parameters are updated based on the number of bytes occupied by each first target field to be migrated. After the parameter configuration is completed, data migration from the source database to the target database is performed. The first target field can be a large object field. By using the storage parameters of the large object field to configure the data migration configuration parameters, such as memory size, the amount of data migrated in a single data migration task can be less than the amount of data limited by the data migration configuration parameters. Therefore, this at least partially overcomes the memory overflow problem caused by the presence of large object fields in the migrated data during the data migration process, better ensuring the security and integrity of the data migration process.

[0067] The following is for reference. Figures 3-4 In conjunction with specific embodiments, Figure 2 The method shown will be further explained.

[0068] Figure 3 A flowchart illustrating the determination of a first target field according to an embodiment of this disclosure is shown schematically.

[0069] like Figure 3 As shown, the process of determining the first target field may include operations S301 to S302.

[0070] Operation S301, in response to a data migration request, uses the system view of the source database to determine the field types of the various data fields recorded in the data table to be migrated.

[0071] Operation S302 determines the first target field that meets preset conditions from multiple data fields, wherein the preset conditions include the field type of the data field being a preset field type.

[0072] According to embodiments of this disclosure, the first target field can be a large object field, which may include: binary large object data (BLOB), character large object data (CLOB), national language character set large object data (NCLOB), and external binary file (BFILE). That is, the preset field type can be binary large object data (BLOB), character large object data (CLOB), national language character set large object data (NCLOB), and external binary file (BFILE).

[0073] According to embodiments of this disclosure, a system view of the source database is used to query fields belonging to a preset field type from all fields as first target fields and perform statistics. Finally, a data record table of the first target field in each data table to be migrated is output, wherein each data record table contains all the first target fields and their related information in the corresponding data table to be migrated.

[0074] According to embodiments of this disclosure, when performing a query on the first target field, the data can be sorted by grouping and sorting according to the data table to be migrated and the field name before filtering.

[0075] According to embodiments of this disclosure, the data record table can be recorded in the form of [table name: field name].

[0076] According to embodiments of this disclosure, the system view of the source database can be dba_lobs.

[0077] According to the embodiments of this disclosure, by using a system view of the source database to determine all field types and then determining the first target field based on a preset field type, the first target field of each data table to be migrated can be obtained quickly and comprehensively.

[0078] Figure 4 A flowchart illustrating the data migration configuration parameter update process according to an embodiment of the present disclosure is shown.

[0079] like Figure 4 As shown, the data migration configuration parameter update process may include operations S401 to S404.

[0080] Operation S401 determines the first byte count information based on the number of bytes occupied by the first target field.

[0081] Operation S402 determines the second byte count information based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the data table to be migrated.

[0082] Operation S403 determines the third byte number information based on the second byte number information.

[0083] Operation S404 updates the data migration configuration parameters using the first byte count information, the second byte count information, and the third byte count information, respectively.

[0084] According to embodiments of this disclosure, the first byte count information can be determined using the number of bytes occupied by the first target field in all data tables to be migrated.

[0085] According to embodiments of this disclosure, the number of bytes occupied by the first target field in each data table to be migrated and the dependency relationship between each first target field and each data table to be migrated can be used to determine the second byte count information.

[0086] According to embodiments of this disclosure, the third byte number information can be obtained by calculating using the second byte number information and the fixed parameter values ​​of the data migration tool.

[0087] According to embodiments of this disclosure, the number of bytes occupied by each first target field is stored, and the storage method is not limited. For example, it can be stored in a table, as shown in Table 1.

[0088] Table 1

[0089] Table name field name byte count MAG_PARAMETER_AREA ta200314 10G MAG_PARAMETER_AREA ta200313 5G MAG_PARAMETER_AREA ta200312 2G DEBT_ACCOUNT def_lob 5G DEBT_ACCOUNT log_lob 2G ASSET_DUEBILL_COMP QUERY_RESULT 10G NPL_ASSET_CLS SESSION_HIST_LOG 10G

[0090] According to embodiments of this disclosure, by utilizing the number of bytes occupied by the first target field to determine the update data migration configuration parameters, the problem of memory overflow caused by the first target object, i.e., the large object, being too large in bytes can be avoided to a certain extent.

[0091] According to an embodiment of this disclosure, operation S401, determining the first byte count information based on the number of bytes occupied by the first target field, includes:

[0092] Based on the number of bytes occupied by the first target field, determine the second target field from the first target field; based on the number of bytes occupied by the second target field, determine the first byte count information.

[0093] According to embodiments of this disclosure, a second target field is determined from the number of bytes occupied by each first target field in each data table to be migrated, wherein the second target field is the first target field that occupies the most bytes in all data tables to be migrated.

[0094] According to an embodiment of this disclosure, a first preset calculation is performed on the second target field to obtain the first byte count information. The first preset calculation may be to multiply the first target byte count by a preset multiple, where the preset multiple may be 1.1.

[0095] According to embodiments of this disclosure, by determining the first target field that occupies the most bytes in all data tables to be migrated based on the number of bytes occupied by the first target field, the upper limit of storage memory from the perspective of the field can be obtained, which helps to avoid the problem of memory overflow.

[0096] According to embodiments of this disclosure, operation S402, determining the second byte count information based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the data table to be migrated, may include the following operations:

[0097] Based on the hierarchical relationship between the first target field and the data table to be migrated, the first target field is divided into at least one field set, and the field set corresponds to the data table to be migrated; for each field set, a third target field is determined from the first target fields included in the field set; and based on the number of bytes occupied by the third target field in each of the at least one field set, the second byte count information is determined.

[0098] According to embodiments of this disclosure, the field set can be divided by assigning one field set to one data table to be migrated based on the dependency relationship between the first target field and the data table to be migrated.

[0099] According to embodiments of this disclosure, a first target field that occupies the most bytes in each field set is determined and used as the third target field.

[0100] According to the embodiments of this disclosure, the number of target bytes occupied by each third target field is determined, and after summing the number of target bytes of all field sets, a second preset calculation is performed to obtain the second byte number information. The second preset calculation can be to multiply the sum of multiple second target byte numbers by a preset multiple, which can be 1.1.

[0101] According to embodiments of this disclosure, by calculating the first target field that occupies the most bytes in each field set, and summing the target byte count of the target field, a second preset calculation is performed. The second byte count information can be used to modify the virtual machine specifications, which helps to avoid memory overflow problems during subsequent data migration.

[0102] According to embodiments of this disclosure, operation S403, determining the third byte count information based on the second byte count information, may include the following operations:

[0103] Based on the fixed parameters of the data migration tool and the second byte number information, the fourth byte number information is determined; and the fourth byte number information is rounded to obtain the third byte number information.

[0104] According to an embodiment of this disclosure, the second byte number information is pre-calculated using the fixed parameters of the data migration tool to obtain the fourth byte number information. Then, the fourth byte number information is rounded to obtain the third byte number information. The fixed parameters can be the upper limit parameter of how much data the data migration tool can pull at one time, and the pre-calculation can be a division operation between the upper limit parameter and the second byte number information. The upper limit parameter can be 10000.

[0105] According to embodiments of this disclosure, by utilizing the second byte number information, the parameter values ​​most suitable for the data migration tool to pull data after modification can be obtained, thereby enabling the use of more suitable parameter values ​​to pull data during data migration.

[0106] According to embodiments of this disclosure, the data migration configuration parameters include process configuration parameters, environment configuration parameters, and task configuration parameters. Operation S404, which updates the data migration configuration parameters using the first byte count information, the second byte count information, and the third byte count information respectively, may include the following operations:

[0107] Using the first preset command, based on the user authentication information included in the data migration request, log in to the business node associated with the data migration tool; using the first byte number information, update the process configuration parameters of the business node; using the second preset command, based on the second byte number information, update the environment configuration parameters of the virtual machine associated with the business node; using the third byte number information, update the task configuration parameters of the data migration task.

[0108] According to embodiments of this disclosure, logging into a business node associated with a data migration tool using a first preset command based on user authentication information included in a data migration request includes: firstly, executing a command to request and invoke the data migration tool; secondly, generating a login script path based on the user authentication information included in the data migration request; and thirdly, connecting a virtual server to the login script path; and finally, executing a command to log the virtual server into the container to jump to the business node. The command to request and invoke the data migration tool can be curl -x, and the command to log the virtual server into the container can be sh connNode.shconnect.

[0109] According to embodiments of this disclosure, updating the environment configuration parameters of a virtual machine associated with a service node using a second preset command based on second byte count information may include: using a web access and parameter passing command to pass the user password and second byte count information, which are then used to dynamically receive and adjust the second byte count information. The web access and parameter passing command can be `curl-d`, and the command used for dynamic adjustment can also be `curl-d`.

[0110] According to embodiments of this disclosure, updating the task configuration parameters of a data migration task using the third byte number information may include: using the curl -d command to modify the value of the task configuration parameters to the value of the third byte number information, and then passing it to the data migration tool management platform.

[0111] According to the embodiments of this disclosure, modifying the process configuration parameters of the business node using the first byte number information helps to avoid memory overflow from the perspective of the business node. At the same time, updating the environment configuration parameters of the virtual machine associated with the business node using the second byte number information helps to avoid memory overflow of migration data due to the virtual machine specification parameters being set too small from an overall perspective. Finally, updating the task configuration parameters of the data migration task using the third byte number information takes the memory information of the migration data into account from the perspective of task configuration. Therefore, memory overflow is prevented from three aspects, ensuring the security of the migration process and the integrity of the migration data during subsequent data migration.

[0112] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0113] Figure 5 A block diagram of a data migration apparatus according to an embodiment of the present disclosure is shown schematically.

[0114] like Figure 5 As shown, the data migration device includes a first target field determination module 510, a byte count determination module 520, a configuration parameter update module 530, and a data migration module 540.

[0115] The first target field determination module 510 is used to determine the first target field from the data table to be migrated recorded in the source database in response to a data migration request.

[0116] The byte count determination module 520 is used to determine the number of bytes occupied by the first target field based on the data stored in the first target field.

[0117] The configuration parameter update module 530 is used to update the data migration configuration parameters based on the number of bytes occupied by the first target field and the subordinate relationship between the first target field and the data table to be migrated.

[0118] The data migration module 540 is used to execute the data migration task included in the data migration request, so as to use the data migration tool to migrate the data to be migrated recorded in the source database to the target database, wherein the data to be migrated includes the data stored in the data table to be migrated.

[0119] According to embodiments of this disclosure, the first target field determination module 510 specifically includes: a field type determination submodule and a target field determination submodule.

[0120] The Field Type Determination submodule is used to determine the field types of multiple data fields in the records of the data table to be migrated in response to a data migration request, using the system view of the source database.

[0121] The target field determination submodule is used to determine the first target field that meets preset conditions from multiple data fields. The preset conditions include that the field type of the data field is a preset field type.

[0122] According to embodiments of this disclosure, the configuration parameter update module 530 specifically includes: a first byte count information determination submodule, a second byte count information determination submodule, a third byte count information determination submodule, and a parameter update submodule.

[0123] The first byte count information determination submodule is used to determine the first byte count information based on the number of bytes occupied by the first target field.

[0124] The second byte count information determination submodule is used to determine the second byte count information based on the number of bytes occupied by the first target field and the subordinate relationship between the first target field and the data table to be migrated.

[0125] The third byte count information determination submodule is used to determine the third byte count information based on the second byte count information.

[0126] The parameter update submodule is used to update the data migration configuration parameters using the first byte count information, the second byte count information, and the third byte count information, respectively.

[0127] According to embodiments of this disclosure, the first byte count information determination submodule specifically includes: a second target field determination unit and a first node information determination unit.

[0128] The second target field determination unit is used to determine the second target field from the first target field based on the number of bytes occupied by the first target field.

[0129] The first node information determination unit is used to determine the first byte number information based on the number of bytes occupied by the second target field.

[0130] According to embodiments of this disclosure, the second byte count information determination submodule specifically includes: a field set partitioning unit, a third target field determination unit, and a second byte count information determination unit.

[0131] The field set partitioning unit is used to divide the first target field into at least one field set based on the dependency relationship between the first target field and the data table to be migrated, with each field set corresponding to the data table to be migrated.

[0132] The third target field determination unit is used to determine the third target field from the first target fields included in each field set.

[0133] The second byte count information determination unit is used to determine the second byte count information based on the number of bytes occupied by the third target field of each of at least one field set.

[0134] According to embodiments of this disclosure, the third byte count information determination submodule includes: a fourth byte count information determination unit and a third byte count acquisition unit.

[0135] The fourth byte number information determination unit is used to determine the fourth byte number information based on the fixed parameters of the data migration tool and the second byte number information.

[0136] The third byte number acquisition unit is used to round the fourth byte number information to obtain the third byte number information.

[0137] According to embodiments of this disclosure, the data migration configuration parameters include process configuration parameters, environment configuration parameters, and task configuration parameters. Specifically, the parameter update submodule includes: a business node login unit, a process configuration parameter determination unit, an environment configuration parameter update unit, and a task configuration parameter update unit.

[0138] The business node login unit is used to log in to the business node associated with the data migration tool using a first preset command based on the user authentication information included in the data migration request.

[0139] The process configuration parameter determination unit is used to update the process configuration parameters of the business node using the first byte number information.

[0140] The environment configuration parameter update unit is used to update the environment configuration parameters of the virtual machine associated with the business node based on the second byte number information using a second preset command.

[0141] The task configuration parameter update unit is used to update the task configuration parameters of the data migration task using the third byte number information.

[0142] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0143] For example, any multiple of the first target field determination module 510, byte count determination module 520, configuration parameter update module 530, and data migration module 540 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first target field determination module 510, byte count determination module 520, configuration parameter update module 530, and data migration module 540 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first target field determination module 510, byte count determination module 520, configuration parameter update module 530, and data migration module 540 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0144] It should be noted that the data migration device part in the embodiments of this disclosure corresponds to the data migration method part in the embodiments of this disclosure. The specific description of the data migration device part is referred to in the data migration method part, and will not be repeated here.

[0145] Figure 6 A block diagram of an electronic device suitable for implementing a data migration method according to an embodiment of the present disclosure is shown schematically. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0146] like Figure 6 As shown, a computer electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0147] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0148] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The electronic device 600 may also include one or more of the following components connected to the I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0149] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0150] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0151] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0152] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0153] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the data migration methods provided in the embodiments of this disclosure.

[0154] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0155] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0156] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0157] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0158] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data migration method, comprising: determining a first target field from a to-be-migrated data table recorded in a source database in response to a data migration request; determining a number of bytes occupied by the first target field based on data stored in the first target field; updating a data migration configuration parameter based on the number of bytes occupied by the first target field and a dependency relationship between the first target field and the to-be-migrated data table, comprising: determining first byte number information based on the number of bytes occupied by the first target field; determining second byte number information based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the to-be-migrated data table, comprising: dividing the first target field into at least one field set based on the dependency relationship between the first target field and the to-be-migrated data table, the field set corresponding to the to-be-migrated data table; determining a third target field from the first target field included in each of the field set; and determining the second byte number information based on the number of bytes occupied by the third target field of each of the at least one field set; and determining third byte number information based on the second byte number information, comprising: determining fourth byte number information based on a fixed parameter of a data migration tool and the second byte number information; and performing an integer processing on the fourth byte number information to obtain the third byte number information; and updating the data migration configuration parameter by using the first byte number information, the second byte number information and the third byte number information respectively, comprising: updating a process configuration parameter of a business node by using the first byte number information, updating an environment configuration parameter of a virtual machine associated with the business node by using the second byte number information, and updating a task configuration parameter of a data migration task by using the third byte number information; and executing a data migration task included in the data migration request to migrate to-be-migrated data recorded in the source database to a target database by using a data migration tool, wherein the to-be-migrated data comprises data stored in the to-be-migrated data table.

2. The method of claim 1, wherein, The determining the first target field from the to-be-migrated data table recorded in the source database in response to the data migration request, comprises: determining a field type of each of a plurality of data fields recorded in the to-be-migrated data table by using a system view of the source database in response to the data migration request; and determining the first target field satisfying a preset condition from the plurality of data fields, wherein the preset condition comprises that the field type of the data field is a preset field type. 3.The method of claim 1, further comprising: logging in a business node associated with the data migration tool based on user authentication information included in the data migration request by using a first preset command, so as to update the data migration configuration parameter by using the first byte number information, the second byte number information and the third byte number information respectively.

4. The method of claim 1, wherein, The determining the first byte number information based on the number of bytes occupied by the first target field, comprises: determine a second target field from the first target field based on a number of bytes occupied by the first target field, wherein the second target field is a first target field occupying a largest number of bytes among all the to-be-migrated data tables; determine the first byte number information based on the number of bytes occupied by the second target field.

5. A data migration apparatus, comprising: a first target field determination module configured to determine a first target field from a to-be-migrated data table recorded in a source database in response to a data migration request; a byte number determination module configured to determine a number of bytes occupied by the first target field based on data stored in the first target field; a configuration parameter update module configured to update a data migration configuration parameter based on the number of bytes occupied by the first target field and a dependency relationship between the first target field and the to-be-migrated data table, including: determining first byte number information based on the number of bytes occupied by the first target field; determining second byte number information based on the number of bytes occupied by the first target field and the dependency relationship between the first target field and the to-be-migrated data table, including: dividing the first target field into at least one field set based on the dependency relationship between the first target field and the to-be-migrated data table, wherein the field set corresponds to the to-be-migrated data table; determining a third target field from the first target field included in the field set for each of the field sets; and determining the second byte number information based on the number of bytes occupied by the third target field of each of the at least one field set; and determining third byte number information based on the second byte number information, including: determining fourth byte number information based on a fixed parameter of a data migration tool and the second byte number information; and performing an integer processing on the fourth byte number information to obtain the third byte number information; and updating the data migration configuration parameter using the first byte number information, the second byte number information and the third byte number information respectively, including: updating a process configuration parameter of a business node using the first byte number information, updating an environment configuration parameter of a virtual machine associated with the business node using the second byte number information, and updating a task configuration parameter of a data migration task using the third byte number information; and a data migration module configured to execute a data migration task included in the data migration request to migrate to-be-migrated data recorded in the source database to a target database using a data migration tool, wherein the to-be-migrated data includes data stored in the to-be-migrated data table.

6. An electronic device, comprising: one or more processors; a memory configured to store one or more instructions, wherein the one or more instructions, when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1 to 4.

7. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to implement the method of any one of claims 1 to 4.

8. A computer program product comprising executable instructions that, when executed by a processor, cause the processor to implement the method of any one of claims 1 to 4.

Images