Data migration method, device, computer equipment and storage medium

By creating temporary data tables during data migration and maintaining the continuity of data access addresses, the problem of task stagnation during migration in traditional technology is solved, and the efficiency of data processing tasks is improved.

CN110245125BInactive Publication Date: 2024-11-01CHINA PING AN LIFE INSURANCE CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN201910418940.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2019-05-20
Publication Date
2024-11-01
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Traditional big data migration technology needs to stop data access during the migration process, resulting in reduced efficiency of data processing tasks for online services.

Method used

By scanning data processing tasks in running state, querying the dependent task data table, creating temporary data tables, pointing the data access address to the temporary data table, and keeping the task running normally during the migration process until the migration is completed, pointing the address to the migration after the migration is completed The following data table.

Benefits of technology

It realizes that the execution of data processing tasks is not interrupted during the data migration process, and improves the efficiency of data processing tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN110245125B_ABST
    Figure CN110245125B_ABST
Patent Text Reader

Abstract

The present application relates to a data migration method, device, computer equipment and storage medium. The method comprises: scanning a data processing task in operation; querying a task data table on which the data processing task depends; creating a temporary data table corresponding to the queried task data table; pointing the data access address in the data processing task to the temporary data table; performing data migration on the task data table; and when the migration of the task data table is completed, pointing the data access address in the data processing task to the migrated task data table. The use of this method enables the data processing task to be executed without interruption during the migration of the task data table, thereby improving the data processing efficiency of the data processing task.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data migration method, device, computer equipment and storage medium. Background Art

[0002] With the development of computer technology and Internet technology, various businesses are processed by computers and transmitted through the Internet, which requires the storage of large amounts of data. The number of databases storing data also increases and decreases accordingly. Sometimes, according to business needs, it is necessary to migrate data in the database between databases, and related technologies for big data migration have emerged.

[0003] However, traditional big data migration technology requires stopping access to data during the migration process in order to migrate the data in the database. Data access can only be continued after the data migration in the database is completed, which will affect the data processing tasks of online businesses and reduce the data processing efficiency of data processing tasks. Summary of the invention

[0004] Based on this, it is necessary to provide a data migration method, device, computer equipment and storage medium that can improve the data processing efficiency of data processing tasks in response to the above technical problems.

[0005] A data migration method, the method comprising:

[0006] Scanning data processing tasks in operation;

[0007] Querying the task data table on which the data processing task depends;

[0008] Create a temporary data table corresponding to the queried task data table;

[0009] Pointing the data access address in the data processing task to the temporary data table;

[0010] Performing data migration on the task data table;

[0011] When the migration of the task data table is completed, the data access address in the data processing task is pointed to the migrated task data table.

[0012] A data migration device, comprising:

[0013] A task scanning module is used to scan the data processing tasks in operation;

[0014] A data table query module, used to query the task data table on which the data processing task depends;

[0015] A data table creation module is used to create a temporary data table corresponding to the queried task data table;

[0016] A pointing adjustment module, used for pointing the data access address in the data processing task to the temporary data table;

[0017] A data migration module, used for performing data migration on the task data table;

[0018] The migration monitoring module is used to point the data access address in the data processing task to the migrated task data table when the migration of the task data table is completed.

[0019] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0020] Scanning data processing tasks in operation;

[0021] Querying the task data table on which the data processing task depends;

[0022] Create a temporary data table corresponding to the queried task data table;

[0023] Pointing the data access address in the data processing task to the temporary data table;

[0024] Performing data migration on the task data table;

[0025] When the migration of the task data table is completed, the data access address in the data processing task is pointed to the migrated task data table.

[0026] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0027] Scanning data processing tasks in operation;

[0028] Querying the task data table on which the data processing task depends;

[0029] Create a temporary data table corresponding to the queried task data table;

[0030] Pointing the data access address in the data processing task to the temporary data table;

[0031] Performing data migration on the task data table;

[0032] When the migration of the task data table is completed, the data access address in the data processing task is pointed to the migrated task data table.

[0033] The above-mentioned data migration method, device, computer equipment and storage medium query the task data table that the data processing task in operation depends on, create a temporary data table corresponding to the task data table, and point the data access address in the data processing task in operation to the temporary data table, so as to ensure that the data processing task can run normally when the task data table is migrated. When the migration of the task data table is completed, the data access address in the data processing task is pointed to the migrated task data table, and at the same time, it is ensured that when the migration of the task data table is completed, the data processing task can continue to run by relying on the migrated task data table, so that during the migration of the task data table, the data processing task is not interrupted, and the data processing efficiency of the data processing task is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 An application scenario diagram of a data migration method in an embodiment;

[0035] Figure 2 A schematic diagram of a data migration method in one embodiment;

[0036] Figure 3 A schematic diagram of a flow chart of steps for confirming storage space in one embodiment;

[0037] Figure 4 A flowchart of steps for screening data processing tasks in one embodiment;

[0038] Figure 5 A flowchart of steps for migrating a task data table in one embodiment;

[0039] Figure 6 A flowchart of steps for generating a data migration report in one embodiment;

[0040] Figure 7 A flowchart of steps for configuring access rights in one embodiment;

[0041] Figure 8 is a structural block diagram of a data migration device in one embodiment;

[0042] Fig. 9 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0044] The data migration method provided in this application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network, the server 104 communicates with the server 106 through the network, the server 106 communicates with the server 108 through the network, and the server 108 communicates with the server 104 through the network. Among them, the terminal 102 can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104, server 106 and server 108 can be implemented by independent servers or server clusters composed of multiple servers. The server 104, server 106 and server 108 can be integrated into the same server.

[0045] In one embodiment, Figure 2 As shown, a data migration method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0046] S202, scanning the data processing tasks in operation.

[0047] Specifically, the server receives the data migration request, and in response to the data migration request, scans the data processing tasks, and determines the data processing tasks in the running state through the scanning. The data processing tasks in the running state are tasks that are currently processing data.

[0048] In one embodiment, a data migration page is displayed in the terminal, and when a click operation on a request trigger button in the data migration page is detected, the data migration parameters input in the data migration page are obtained, a data migration request is generated according to the data migration parameters, and the data migration request is sent to the server. The server receives the data migration request sent by the terminal.

[0049] S204, querying the task data table on which the data processing task depends.

[0050] Specifically, the server extracts the task identifier of the data processing task, queries the data processing information corresponding to each task identifier, extracts the data table identifier from the data processing information, and the task data table corresponding to the extracted data table identifier is the task data table that the data processing task depends on. The task data table that the data processing task depends on is the task data table that needs to be accessed to execute the data processing task.

[0051] S206: Create a temporary data table corresponding to the queried task data table.

[0052] Specifically, when the server queries the task data table that the data processing task depends on, it copies the task data table that each data processing task depends on, and uses the copied data table as the temporary data table corresponding to the task data table, thereby obtaining the temporary data table corresponding to each task data table.

[0053] In one embodiment, after querying the data processing task data table, the server reads the data in each data processing task table, adds the read data to a pre-created data table, and generates a temporary data table corresponding to the task data table.

[0054] S208, pointing the data access address in the data processing task to the temporary data table.

[0055] Specifically, when the server creates a temporary data table, it records the access address of the temporary data table. The server parses the data processing task, determines the data access address in the data processing task through parsing, replaces the determined data access address with the access address of the temporary data table, and changes the data access address in the data processing task so that the changed data access address in the data processing task points to the temporary data table.

[0056] In one embodiment, when the server creates a temporary data table, the access address of the temporary data table is stored in correspondence with the data access address of the corresponding task data table. After determining the data access address in the data processing task, the server queries the access address of the temporary data table corresponding to the determined data access address, and modifies the data access address in the data processing task to obtain the access address corresponding to the queried temporary data table.

[0057] S210, performing data migration on the task data table.

[0058] Specifically, the server obtains the address of the migration-in database, establishes a connection with the migration-in database according to the address of the migration-in database, and sends the task data table to the migration-in database through the established connection.

[0059] In one embodiment, the server obtains a data table identifier and an address of a migration-in database of a task data table, creates a migration task according to the obtained data table identifier and the address of the migration-in database, and builds a migration task queue according to the created migration task. The server reads the migration task from the migration task queue, queries the task data table according to the data table identifier in the migration task queue, and migrates the read task data table into the migration-in database corresponding to the address of the migration-in database in the migration task.

[0060] S212, when the task data table is migrated, the data access address in the data processing task is pointed to the migrated task data table.

[0061] Specifically, when the server migrates the task data table, it monitors the migration of the task data table. When it detects that the migration of the task data table is completed, it obtains the migration access address corresponding to the migrated task data table, and modifies the data access address in the data processing task to the migration access address, so that the data access address in the data processing task points to the migrated task data table.

[0062] In one embodiment, the server monitors the migration task queue, and when it is monitored that the migration task in the migration task queue is completely read, it is monitored that the migration of the task data table is completed.

[0063] In this embodiment, the task data table that the data processing task in the running state depends on is queried, a temporary data table corresponding to the task data table is created, and the data access address in the data processing task in the running state is pointed to the temporary data table, so as to ensure that the data processing task can run normally when the task data table is migrated. When the migration of the task data table is completed, the data access address in the data processing task is pointed to the migrated task data table, and at the same time, it is ensured that when the migration of the task data table is completed, the data processing task can continue to run by relying on the migrated task data table, so that during the migration of the task data table, the data processing task is not interrupted, and the data processing efficiency of the data processing task is improved.

[0064] like Figure 3 As shown, in one embodiment, before S202, a step of confirming the storage space is also specifically included, and the step specifically includes the following contents:

[0065] S302: Obtain a data migration instruction.

[0066] Specifically, the terminal obtains the set migration-out database address and migration-in database address, encapsulates the obtained migration-out database address and migration-in database address, generates a data migration instruction by encapsulation, and sends the data migration instruction to the server 104 in the big data platform. The big data platform is composed of at least server 104, server 106, and server 106.

[0067] S304, extracting the migration-out database address and the migration-in database address in the data migration instruction.

[0068] Specifically, after receiving the data migration instruction, the server 104 parses the data migration instruction and extracts the migration-out database address and the migration-in database address from the data migration instruction through parsing. The migration-out database address is the address of the migration-out database where the task data table is currently located, and the migration-out database can be set in the server 106. The migration-in database address is the address of the migration-in database to which the task data table is to be migrated, and the migration-in database can be set in the server 108.

[0069] S306: Count the storage space capacity in the migration-out database address.

[0070] Specifically, the server 104 generates a used capacity query request according to the address of the migration database, and sends the used capacity query request to the server 106. The server 106 parses the used capacity query request, extracts the migration database address in the used capacity query request through parsing, counts the storage space capacity in the migration database corresponding to the migration database address, and returns the stored space capacity to the server 104. The server 104 receives the storage space capacity returned by the server 106.

[0071] S308, determining whether the remaining space capacity corresponding to the migration database address is greater than the stored space capacity.

[0072] Specifically, the server 104 generates a remaining capacity query request according to the address of the outgoing database, and sends the remaining capacity query request to the server 108. The server 108 parses the remaining capacity query request, extracts the address of the incoming database in the remaining capacity query request through parsing, counts the remaining space capacity in the incoming database corresponding to the address of the incoming database, and returns the remaining space capacity to the server 104. The server 104 receives the remaining space capacity returned by the server 106. The server 104 compares the remaining space capacity of the incoming database with the stored space capacity of the outgoing database to determine whether the remaining space capacity is greater than the stored space capacity.

[0073] S310, when the remaining space capacity is greater than the stored space capacity, scanning the data processing tasks in operation.

[0074] Specifically, when the remaining space capacity is greater than the stored space capacity, it indicates that there is enough storage space in the migration-in database to store the task data table migrated out of the migration-out database; when the remaining space capacity is less than or equal to the stored space capacity, it indicates that there is not enough storage space in the migration-in database to store the task data table migrated out of the migration-out database. When the remaining space capacity is greater than the stored space capacity, the data processing tasks in operation are scanned.

[0075] In this embodiment, after obtaining the data migration instruction, the stored space capacity in the migration-out database and the corresponding remaining space capacity in the migration-in database are counted. When the remaining space capacity is greater than the stored space capacity, the processing tasks in operation are scanned to ensure that there is sufficient remaining space capacity in the migration-in database, and then the data migration is started, thereby avoiding data migration anomalies caused by insufficient storage space during the data migration process.

[0076] like Figure 4As shown, in one embodiment, S202 specifically includes a step of screening data processing tasks, which specifically includes the following contents:

[0077] S402, scanning the task identifier corresponding to the data processing task.

[0078] Specifically, the server 104 scans the data processing task and determines the task identifier corresponding to the data processing task through the scanning. The task identifier is used to uniquely identify the data processing task.

[0079] S404: Determine the working status corresponding to each task identifier.

[0080] Specifically, after the server 104 scans the task identifier corresponding to the data processing task, it extracts the state identifier corresponding to each task identifier and identifies the working state corresponding to the state identifier corresponding to each task identifier.

[0081] S406, filtering data processing tasks corresponding to task identifiers whose working status is running status.

[0082] Specifically, the server 104 identifies a task identifier whose working status is a running status, determines a task identifier corresponding to the status identifier of the running status, and screens a data processing task corresponding to the determined task identifier.

[0083] In this embodiment, the data processing task corresponding to the task identifier in the running state is determined according to the working state corresponding to the task identifier, so as to determine the data processing task in the running state that needs to be migrated, thereby ensuring accurate migration of the data processing task in the running state.

[0084] like Figure 5 As shown, in one embodiment, S210 specifically includes a step of migrating a task data table, which specifically includes the following contents:

[0085] S502: Create a data migration task according to the task data table.

[0086] Specifically, the server obtains a data table identifier of the task data table, creates a data migration task corresponding to the task data table according to the obtained data table identifier, and obtains a data migration task corresponding to each task data table. The data migration task is a task for migrating the task data table.

[0087] S504: Add the created data migration task to a preset number of migration task queues.

[0088] Specifically, a preset number of migration task queues are set in the server. The server adds the data migration tasks corresponding to the created task data tables to the migration task queues to obtain the migration task queues including the data migration tasks.

[0089] S506: Read the data migration tasks in each migration task queue in parallel.

[0090] Specifically, the server processes the data migration tasks in each migration task queue in parallel and reads the data migration tasks in each migration task queue in parallel.

[0091] S508: Perform data migration on the task data table corresponding to the read data migration task.

[0092] Specifically, the server parses each data migration task read, extracts the task table identifier in the data migration task through parsing, queries the task data table corresponding to the task table identifier, obtains the migration database address, and stores the queried task data table in the database corresponding to the migration database address.

[0093] In this embodiment, a data migration task is created according to the task data table, and the created data migration task is added to a preset number of migration task queues. The data migration tasks in each migration task queue are read in parallel and processed in parallel to avoid network congestion due to too many data migration tasks, thereby reducing the migration efficiency of the task data table.

[0094] like Figure 6 As shown, in one embodiment, after S210, the step of generating a data migration report is further included, and the step specifically includes the following contents:

[0095] S602, recording the migration process data of the task data table.

[0096] Specifically, when migrating the task data table, the server records the migration process data of each task data table. The migration process data includes at least one of the migration status, migration start time, migration end time and migration time of the task data table.

[0097] S604: Generate a data migration report according to the recorded migration process data.

[0098] Specifically, after recording the migration process data, the server stores the data table identifier of the task data table and the migration process data in correspondence, and generates a data migration report.

[0099] S606, sending the data migration report to the terminal logged in with the administrator account.

[0100] Specifically, after generating the data migration report, the server obtains the administrator account, queries the terminal address corresponding to the administrator account, and sends the data migration report to the terminal corresponding to the queried terminal address.

[0101] In this embodiment, during the migration process of the task data table, the migration process data of the task data table is recorded, a data migration report is generated based on the migration process data, the migration status of the data migration process is recorded through the data migration report, and the data migration report is sent to the terminal logged in by the administrator account to ensure that the administrator can supervise the data through the data migration report.

[0102] like Figure 7 As shown, in one embodiment, after S212, a step of configuring access rights is specifically included, and the step specifically includes the following contents:

[0103] S702, obtaining access permission information corresponding to the task data table.

[0104] Specifically, the server statically obtains the data table identifier of each task data table, generates an access permission acquisition request corresponding to the data table identifier, sends the access permission acquisition request to the database corresponding to the migration database address, and obtains the access permission information corresponding to the data table identifier queried by the migration database according to the access permission acquisition request.

[0105] S704, determining the migrated task data table corresponding to the task data table.

[0106] Specifically, after migrating the task data table, the server obtains the data table address of the migrated task data table, and stores the migrated data table address in correspondence with the data table identifier of the task data table. The server queries the data table address corresponding to the data table identifier, and the queried data table address is the migrated data table address corresponding to the task data table.

[0107] S706: Configure access permissions for the migrated data table address according to the access permission information.

[0108] Specifically, after migrating the task data table to the migration-in database corresponding to the migration-in database address, the server determines the migrated task data table and configures access rights for the migrated data table address according to the access right information.

[0109] In this embodiment, after the task data table is migrated, the access rights of the migrated data table are configured according to the access right information of the task data table, thereby ensuring access control to the migrated task data table and ensuring the security of the migrated task data table.

[0110] It should be understood that although Figure 2-7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-7At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0111] In one embodiment, Figure 8 As shown, a data migration device 800 is provided, including: a task scanning module 802, a data table query module 804, a data table creation module 806, a pointing adjustment module 808, a data migration module 810 and a migration monitoring module 812, wherein:

[0112] The task scanning module 802 is used to scan the data processing tasks in operation.

[0113] The data table query module 804 is used to query the task data table on which the data processing task depends.

[0114] The data table creation module 806 is used to create a temporary data table corresponding to the queried task data table.

[0115] The pointing adjustment module 808 is used to point the data access address in the data processing task to the temporary data table.

[0116] The data migration module 810 is used to migrate data on the task data table.

[0117] The migration monitoring module 812 is used to point the data access address in the data processing task to the migrated task data table when the migration of the task data table is completed.

[0118] In one embodiment, the data migration device 800 further includes: a capacity confirmation module.

[0119] The capacity confirmation module is used to obtain the data migration instruction; extract the migration-out database address and the migration-in database address in the data migration instruction; count the storage space capacity corresponding to the migration-out database address; and determine whether the remaining space capacity corresponding to the migration-in database address is greater than the storage space capacity;

[0120] The task scanning module 802 is also used to scan the data processing tasks in operation when the remaining space capacity is greater than the stored space capacity.

[0121] In one embodiment, the task scanning module 802 is further used to scan the task identifiers corresponding to the multiple data processing tasks; determine the working status corresponding to the task identifiers; and filter the data processing tasks corresponding to the task identifiers whose working status is the running status.

[0122] In one embodiment, the data migration module 810 is also used to create a data migration task based on the task data table; add the created data migration task to a preset number of migration task queues; read the data migration tasks in each migration task queue in parallel; and perform data migration on the task data table corresponding to the read data migration task.

[0123] In one embodiment, the data migration module 810 is further used to record the migration process data of the task data table; generate a data migration report according to the recorded migration process data; and send the data migration report to a terminal logged in with an administrator account.

[0124] In one embodiment, the data migration module 810 is further used to obtain access permission information corresponding to the task data table; determine the migrated data table address corresponding to the task data table; and configure access permission for the migrated data table address according to the access permission information.

[0125] In this embodiment, the task data table that the data processing task in the running state depends on is queried, a temporary data table corresponding to the task data table is created, and the data access address in the data processing task in the running state is pointed to the temporary data table, so as to ensure that the data processing task can run normally when the task data table is migrated. When the migration of the task data table is completed, the data access address in the data processing task is pointed to the migrated task data table, and at the same time, it is ensured that when the migration of the task data table is completed, the data processing task can continue to run by relying on the migrated task data table, so that during the migration of the task data table, the data processing task is not interrupted, and the data processing efficiency of the data processing task is improved.

[0126] For the specific definition of the data migration device, please refer to the definition of the data migration method above, which will not be repeated here. Each module in the above-mentioned data migration device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0127] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9As shown. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data tables. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data migration method is implemented.

[0128] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: scanning a data processing task in a running state; querying a task data table on which the data processing task depends; creating a temporary data table corresponding to the queried task data table; pointing a data access address in the data processing task to the temporary data table; performing data migration on the task data table; and when the migration of the task data table is completed, pointing a data access address in the data processing task to the migrated task data table.

[0130] In one embodiment, before scanning the data processing tasks in the running state, the processor also implements the following steps when executing the computer program: obtaining data migration instructions; extracting the migration-out database address and the migration-in database address in the data migration instructions; counting the stored space capacity corresponding to the migration-out database address; determining whether the remaining space capacity corresponding to the migration-in database address is greater than the stored space capacity; when the remaining space capacity is greater than the stored space capacity, scanning the data processing tasks in the running state.

[0131] In one embodiment, scanning the data processing tasks in the running state includes: scanning the task identifiers corresponding to the multiple data processing tasks; determining the working state corresponding to the task identifiers; and screening the data processing tasks corresponding to the task identifiers whose working state is the running state.

[0132] In one embodiment, performing data migration on a task data table includes: creating a data migration task according to the task data table; adding the created data migration task to a preset number of migration task queues; reading the data migration tasks in each migration task queue in parallel; and performing data migration on the task data table corresponding to the read data migration task.

[0133] In one embodiment, after performing data migration on the task data table, the processor further implements the following steps when executing the computer program: recording the migration process data of the task data table; generating a data migration report based on the recorded migration process data; and sending the data migration report to a terminal logged in with an administrator account.

[0134] In one embodiment, when the task data table is migrated, after the data access address in the data processing task is pointed to the migrated task data table, the processor also implements the following steps when executing the computer program: obtaining access permission information corresponding to the task data table; determining the migrated data table address corresponding to the task data table; and configuring access permission for the migrated data table address according to the access permission information.

[0135] In this embodiment, the task data table that the data processing task in the running state depends on is queried, a temporary data table corresponding to the task data table is created, and the data access address in the data processing task in the running state is pointed to the temporary data table, so as to ensure that the data processing task can run normally when the task data table is migrated. When the migration of the task data table is completed, the data access address in the data processing task is pointed to the migrated task data table, and at the same time, it is ensured that when the migration of the task data table is completed, the data processing task can continue to run by relying on the migrated task data table, so that during the migration of the task data table, the data processing task is not interrupted, and the data processing efficiency of the data processing task is improved.

[0136] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: scanning a data processing task in a running state; querying a task data table on which the data processing task depends; creating a temporary data table corresponding to the queried task data table; pointing a data access address in the data processing task to the temporary data table; migrating data in the task data table; and when the migration of the task data table is completed, pointing a data access address in the data processing task to the migrated task data table.

[0137] In one embodiment, before scanning the data processing tasks in the running state, the computer program also implements the following steps when executed by the processor: obtaining data migration instructions; extracting the migration-out database address and the migration-in database address in the data migration instructions; counting the stored space capacity corresponding to the migration-out database address; determining whether the remaining space capacity corresponding to the migration-in database address is greater than the stored space capacity; when the remaining space capacity is greater than the stored space capacity, scanning the data processing tasks in the running state.

[0138] In one embodiment, scanning the data processing tasks in the running state includes: scanning the task identifiers corresponding to the multiple data processing tasks; determining the working state corresponding to the task identifiers; and screening the data processing tasks corresponding to the task identifiers whose working state is the running state.

[0139] In one embodiment, performing data migration on a task data table includes: creating a data migration task according to the task data table; adding the created data migration task to a preset number of migration task queues; reading the data migration tasks in each migration task queue in parallel; and performing data migration on the task data table corresponding to the read data migration task.

[0140] In one embodiment, after data migration is performed on the task data table, the computer program further implements the following steps when executed by the processor: recording the migration process data of the task data table; generating a data migration report based on the recorded migration process data; and sending the data migration report to a terminal logged in with an administrator account.

[0141] In one embodiment, when the task data table migration is completed, after the data access address in the data processing task is pointed to the migrated task data table, the computer program also implements the following steps when executed by the processor: obtaining access permission information corresponding to the task data table; determining the migrated data table address corresponding to the task data table; and configuring access permission for the migrated data table address based on the access permission information.

[0142] In this embodiment, the task data table that the data processing task in the running state depends on is queried, a temporary data table corresponding to the task data table is created, and the data access address in the data processing task in the running state is pointed to the temporary data table, so as to ensure that the data processing task can run normally when the task data table is migrated. When the migration of the task data table is completed, the data access address in the data processing task is pointed to the migrated task data table, and at the same time, it is ensured that when the migration of the task data table is completed, the data processing task can continue to run by relying on the migrated task data table, so that during the migration of the task data table, the data processing task is not interrupted, and the data processing efficiency of the data processing task is improved.

[0143] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0144] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data migration method, the method comprising: Get data migration instructions; Extract the outgoing database address and the incoming database address in the data migration instruction; Counting the storage space capacity corresponding to the migration-out database address; Determine whether the remaining space capacity corresponding to the migration database address is greater than the stored space capacity; When the remaining space capacity is greater than the stored space capacity, scanning the data processing tasks in operation; Querying the task data table on which the data processing task depends; Create a temporary data table corresponding to the queried task data table; The data access address of the temporary data table is stored correspondingly to the data access address of the corresponding task data table; Parsing the data access address in the data processing task, and querying the data access address of the temporary data table corresponding to the data access address in the data processing task; Modifying the data access address in the data processing task to the data access address corresponding to the queried temporary data table; Creating a data migration task corresponding to the task data table according to the data table identifier of the task data table, and obtaining a data migration task corresponding to each of the task data tables; Adding each of the created data migration tasks to a preset number of migration task queues; Read the data migration tasks in each migration task queue in parallel; Parse the data table identifier in the read data migration task, query the task data table and the migration-in database address corresponding to the data table identifier in the data migration task, and store the queried task data table in the database corresponding to the migration-in database address; When the migration of the task data table is completed, the data access address in the data processing task is pointed to the migrated task data table.

2. The method according to claim 1, characterized in that The step of pointing the data access address in the data processing task to the migrated task data table includes: Get the migration access address corresponding to the migrated task data table; The data access address in the data processing task is modified to the migration access address.

3. The method according to claim 1, characterized in that The data processing tasks of the scanning in the running state include: Scan task identifiers corresponding to multiple data processing tasks; Determine the working status corresponding to the task identifier; Filter the data processing tasks corresponding to the task IDs whose working status is running.

4. The method according to claim 1, characterized in that: After the data migration of the task data table, the method further includes: Recording the migration process data of the task data table; Generate a data migration report based on the recorded migration process data; The data migration report is sent to a terminal logged in with an administrator account.

5. The method according to claim 4, characterized in that The migration process data includes at least one of the migration status, migration start time, migration end time and migration time consumed of the task data table.

6. The method according to any one of claims 1 to 5, characterized in that When the task data table is migrated, after the data access address in the data processing task is pointed to the migrated task data table, the method further includes: Obtaining access permission information corresponding to the task data table; Determine the migrated data table address corresponding to the task data table; The access permission is configured for the migrated data table address according to the access permission information.

7. A data migration device, characterized in that: The device comprises: The capacity confirmation module is used to obtain the data migration instruction; extract the migration-out database address and the migration-in database address in the data migration instruction; count the stored space capacity corresponding to the migration-out database address; and determine whether the remaining space capacity corresponding to the migration-in database address is greater than the stored space capacity; A task scanning module, used for scanning the data processing tasks in operation when the remaining space capacity is greater than the stored space capacity; A data table query module, used to query the task data table on which the data processing task depends; A data table creation module is used to create a temporary data table corresponding to the queried task data table; A pointing adjustment module is used to store the data access address of the temporary data table in correspondence with the data access address of the corresponding task data table; parse the data access address in the data processing task, query the data access address of the temporary data table corresponding to the data access address in the data processing task; modify the data access address in the data processing task to the data access address corresponding to the queried temporary data table; A data migration module is used to create a data migration task corresponding to the task data table according to the data table identifier of the task data table, and obtain the data migration tasks corresponding to each of the task data tables; add each of the created data migration tasks to a preset number of migration task queues; read the data migration tasks in each migration task queue in parallel; parse the data table identifier in the read data migration task, query the task data table and the migration-in database address corresponding to the data table identifier in the data migration task, and store the queried task data table in the database corresponding to the migration-in database address; The migration monitoring module is used to point the data access address in the data processing task to the migrated task data table when the migration of the task data table is completed.

8. The device according to claim 7, characterized in that The migration monitoring module is also used for: Get the migration access address corresponding to the migrated task data table; The data access address in the data processing task is modified to the migration access address.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Processing method and apparatus for data migration

    CN106534226A

  • Data migration method and device

    CN106909595A

  • Business processing method and apparatus for online data migration process

    CN108268501A

  • Data migration method and device, server and storage medium

    CN108509556A

  • Migrating temporary data of a session

    US20080098048A1