Method, device, equipment, and storage medium for migrating data

Abstract

This application discloses a method, apparatus, device, and storage medium for migrating data, relating to the field of distributed technology. A specific implementation involves obtaining an identifier for target data; marking the identifier; copying the target data from a source database to a target database; and, upon determining that the target data has been copied, removing the identifier. This implementation provides a data migration solution that can migrate data from a source database to a target database.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the field of distributed technology, and more particularly to a method, apparatus, device, and storage medium for migrating data. Background Art

[0002] In e-commerce, advertising data is stored in multiple tables by identifier. To prevent uncontrolled data growth, each advertiser has their own upper limits for different table dimensions. However, with the continuous increase in new and active advertisers, the data in the material library is growing at an increasingly rapid rate. A single MySQL database is no longer sufficient for storage, and sharding is required to distribute read and write traffic.

[0003] After acquiring the new elastic database resources, the original physical database held a large amount of data, requiring high availability and strong consistency. A one-time migration to the new database presented significant risks, necessitating migration by identification dimension. This migration process was a long, ongoing effort. Furthermore, the continuous iteration of advertising demand created new tables, causing both the old and new databases to maintain newly added tables during the data migration phase. This necessitated a migration solution that could flexibly accommodate the migration of new tables, eliminating the need for repetitive migration development for these newly added tables. Summary of the Invention

[0004] Provided are a method, apparatus, device, and storage medium for migrating data.

[0005] According to a first aspect, a method for migrating data is provided, comprising: obtaining an identifier of target data; marking the identifier; copying the target data from a source database to a target database; and removing the identifier in response to determining that the target data copying is complete.

[0006] According to a second aspect, a device for migrating data is provided, comprising: an acquisition unit configured to acquire an identifier of target data; a marking unit configured to mark the identifier; a copying unit configured to copy the target data from a source database to a target database; and a removal unit configured to remove the marker in response to determining that the target data copying is complete.

[0007] According to a third aspect, an electronic device for migrating data is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the at least one processor executes the method described in the first aspect.

[0008] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided. The computer instructions are used to cause a computer to execute the method described in the first aspect.

[0009] According to a fifth aspect, a computer program product comprises a computer program, wherein the computer program implements the method described in the first aspect when executed by a processor.

[0010] The technology of this application provides a data migration solution that can migrate data from a source database to a target database.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present application.

[0013] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present application may be applied;

[0014] Figure 2 is a flowchart of an embodiment of a method for migrating data according to the present application;

[0015] Figure 3 is a schematic diagram of an application scenario of the method for migrating data according to the present application;

[0016] Figure 4 is a flowchart of another embodiment of a method for migrating data according to the present application;

[0017] Figure 5 is a structural diagram of an embodiment of an apparatus for migrating data according to the present application;

[0018] Figure 6 It is a block diagram of an electronic device used to implement the method for migrating data in an embodiment of the present application. DETAILED DESCRIPTION

[0019] The following description of exemplary embodiments of the present application is made in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0020] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0021] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method for migrating data or the apparatus for migrating data of the present application can be applied.

[0022] like Figure 1 As shown, system architecture 100 may include a source database 101, a server 102, and a target database 103. Source database 101 and server 102 are connected to each other via a network, as well as server 102 and target database 103. The network may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0023] Source database 101 and target database 103 can be used to store data. Source database 101 can store a large amount of data. After receiving a migration instruction from a user, server 102 can migrate the data from source database 101 to target database 103. Source database 101 and target database 103 can be distributed databases, including multiple database shards, each of which can include multiple data tables. Data can be stored in these data tables.

[0024] Server 102 can be hardware or software. When server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When server 105 is software, it can be implemented as multiple software or software modules (for example, to provide distributed services), or it can be implemented as a single software or software module. No specific limitations are given here.

[0025] It should be noted that the method for migrating data provided in the embodiment of the present application is generally executed by the server 102. Accordingly, the device for migrating data is generally provided in the server 102.

[0026] It should be understood that Figure 1 The number of databases and servers in the embodiment is only for illustration. Any number of databases and servers may be provided as required.

[0027] Continue to refer Figure 2 , shows a process 200 of an embodiment of a method for migrating data according to the present application. The method for migrating data in this embodiment includes the following steps:

[0028] Step 201: Obtain the identifier of the target data.

[0029] In this embodiment, the execution subject of the method for migrating data (for example Figure 1 The server 102 shown in FIG. 102 may obtain the identifier of the target data in various ways. For example, the execution entity may receive the identifier of the target data sent by a user via a terminal device. The identifier of the target data may be the identifier of the advertiser, such as the advertiser's name. In some specific applications, the execution entity may add the identifier of the target data to a target queue each time it receives a migration instruction from a user. The execution entity may obtain the identifier of the target data from the target queue. In this way, if the execution entity is a distributed device, the identifier of the target data may be obtained from the target queue separately.

[0030] Step 202: Mark the identifier.

[0031] After obtaining the identifier of the target data, the executing entity can mark it. This acts as a "blacklist" to temporarily suspend the writing of the data indicated by the identifier. This prevents data migration errors caused by data writing during the data migration process. When reading or writing data, the data reader / writer can first check whether the data identifier is marked. If so, it indicates that the data cannot be written at this time.

[0032] Step 203: Copy the target data from the source database to the target database.

[0033] After marking the target data, the execution entity can copy the target data from the source database to the target database. Specifically, the execution entity can execute the above copy task through multiple threads. Each thread can be responsible for copying the data in a single data table.

[0034] Step 204: In response to determining that the target data copying is completed, remove the above mark.

[0035] After the execution entity completes copying the target data, it can remove the aforementioned mark. This allows both the data in the source database and the data migrated to the target database to be read and written by users. It is understood that to ensure data consistency between the source and target databases, read and write logs can be generated during the reading and writing of data in the source and target databases.

[0036] Continue to see Figure 3 , which shows a schematic diagram of an application scenario of the method for migrating data according to the present application. Figure 3 In the application scenario, after receiving the migration instruction sent by the user, server 301 obtains the identifier abc of the target data from source database 302. It then marks identifier abc as non-writable. It then copies the target data to target databases 303 and 303. Finally, after the target data is copied, the "non-writable" mark is removed.

[0037] The method for migrating data provided by the above-mentioned embodiment of the present application can mark data during the data migration process, thereby avoiding data migration errors caused by writing data during the migration process.

[0038] Continue to see Figure 4 , which shows a process 400 of another embodiment of the method for migrating data according to the present application. Figure 4 As shown, the method of this embodiment may include the following steps:

[0039] Step 401: Obtain the identifier of the target data.

[0040] Step 402 : Search the preset data status list for the identifier to verify whether the status corresponding to the identifier is to be migrated. In response to determining that the status corresponding to the identifier is to be migrated, mark the identifier.

[0041] In this embodiment, after obtaining the identifier of the target data, the executing entity may first query a preset data status list, which includes the correspondence between identifiers and statuses. If the status corresponding to the identifier is found to be pending migration, the identifier is marked. If the status corresponding to the identifier is migrated, it means that the data has been migrated, and the identifier of the new data can be obtained.

[0042] Step 403: Copy the target data from the source database to the target database.

[0043] Step 404 : In response to determining that the target data copying is completed, the mark is removed and the data status list is updated.

[0044] If the target data is copied, the execution subject can remove the mark and update the data status list. Specifically, the execution subject can update the state corresponding to the mark in the data status list.

[0045] Step 405 , generating a migration log during the copy process; after the copy is completed, comparing the data copied to the target database with the target data; in response to determining that the two are different, performing a rollback operation according to the migration log and deleting the data copied to the target database.

[0046] In this embodiment, the execution entity can also generate a migration log during the copy process. The migration log includes information such as the data source, the data destination path, and the copy time. After the copy is complete, the execution entity can compare the data copied to the target database with the target data. If the two are different, it indicates that an error occurred during the migration process. The execution entity can then perform a rollback based on the migration log and delete the data copied to the target database.

[0047] In some optional implementations of this embodiment, the migration log includes a migration batch. A migration batch refers to a batch of migrations that are made up of Figure 1 The data that the operator initiates the migration. The above Fangfang may also include Figure 4 The following steps are not shown in the figure: in response to determining that an error occurs in the data in the target database, verifying other data in the same migration batch as the error data.

[0048] In this implementation, if the execution entity determines that data in the target database has errors, it can verify other data in the same migration batch as the erroneous data. The errors here can be various errors, such as read and write errors.

[0049] Step 406 : In response to determining that the two are the same, determine the duration between the target data migration completion time and the current time; in response to determining that the duration is greater than the preset duration, delete the target data in the source database.

[0050] If the two values ​​match, the data migration is successful. The execution entity can then determine the duration between the target data migration completion time and the current time. If the duration exceeds the preset duration, the target data in the source database can be deleted. This saves storage space.

[0051] Step 407 : In response to determining that incremental data of the target data is detected after the copying is completed, the incremental data is copied to the target database; in response to determining that the incremental data copying is completed, the incremental data is assembled with the target data.

[0052] In this embodiment, if the execution entity detects incremental data in the target database after the copy is complete, this indicates that a user has written to data in the source database after the target data was migrated to the target database. Therefore, the incremental data needs to be copied to the target database. Once the incremental data copy is complete, the incremental data can be reassembled with the target data to ensure data consistency.

[0053] The method for migrating data provided in the above embodiment of the present application can verify the data after the migration is completed to ensure the accuracy of the data. It can also migrate incremental data to the target database and assemble the incremental data with the target data after the migration is completed.

[0054] Further references Figure 5 As an implementation of the methods shown in the above figures, the present application provides an embodiment of a device for migrating data. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0055] like Figure 5As shown, the apparatus 500 for migrating data in this embodiment includes: an acquiring unit 501 , a marking unit 502 , a copying unit 503 and a removing unit 504 .

[0056] The acquiring unit 501 is configured to acquire an identifier of target data.

[0057] The marking unit 502 is configured to mark the identifier.

[0058] The copy unit 503 is configured to copy the target data from the source database to the target database.

[0059] The removal unit 504 is configured to remove the mark in response to determining that the target data copying is completed.

[0060] In some optional implementations of this embodiment, the marking unit 502 may be further configured to: query the identifier in the preset data status list to verify whether the status corresponding to the identifier is to be migrated; in response to determining that the status corresponding to the identifier is to be migrated, mark the identifier.

[0061] In some optional implementations of this embodiment, the apparatus 500 may further include: Figure 5 The updating unit not shown in the figure is configured to: in response to determining that the target data copying is completed, update the data status list.

[0062] In some optional implementations of this embodiment, the apparatus 500 may further include: Figure 5 Not shown: generation unit, comparison unit, rollback unit.

[0063] The generation unit is configured to generate a migration log during the replication process.

[0064] The comparison unit is configured to compare the data copied to the target database with the target data after the copying is completed.

[0065] The rollback unit is configured to, in response to determining that the two are different, perform a rollback operation according to the migration log and delete the data copied to the target database.

[0066] In some optional implementations of this embodiment, the migration log includes a migration batch. The apparatus 500 may further include Figure 5 The verification unit (not shown) is configured to: in response to determining that an error occurs in the data in the target database, verify other data in the same migration batch as the erroneous data.

[0067] In some optional implementations of this embodiment, the apparatus 500 may further include: Figure 5The deletion unit not shown in the figure is configured to: in response to determining that the two are the same, determine the duration between the target data migration completion time and the current time; in response to determining that the duration is greater than the preset duration, delete the target data in the source database.

[0068] In some optional implementations of this embodiment, the apparatus 500 may further include: Figure 5 The assembly unit not shown in the figure is configured to: in response to determining that incremental data of the target data is detected after the copying is completed, copy the incremental data to the target database; in response to determining that the incremental data copying is completed, assemble the incremental data with the target data.

[0069] It should be understood that the units 501 to 504 described in the apparatus 500 for migrating data are respectively the same as those in the reference Figure 2 Therefore, the operations and features described above for the method for migrating data are also applicable to the apparatus 500 and the units included therein, and will not be described in detail here.

[0070] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0071] Figure 6 A block diagram of an electronic device 600 for performing a method for migrating data according to an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0072] like Figure 6 As shown, the device 600 includes a processor 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a memory 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The processor 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An I / O interface (input / output interface) 605 is also connected to the bus 604.

[0073] Various components in device 600 are connected to I / O interface 605, including: input unit 606, such as a keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage 608, such as a magnetic disk, optical disk, etc.; and communication unit 609, such as a network card, modem, wireless communication transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0074] The processor 601 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the processor 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 601 performs the various methods and processes described above, such as the method for migrating data. For example, in some embodiments, the method for migrating data can be implemented as a computer software program that is tangibly contained in a machine-readable storage medium, such as the memory 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the processor 601, one or more steps of the method for migrating data described above can be performed. Alternatively, in other embodiments, the processor 601 can be configured to perform the method for migrating data by any other appropriate means (e.g., by means of firmware).

[0075] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0076] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. The above program code can be packaged into a computer program product. These program codes or computer program products can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that the program code, when executed by the processor 601, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or completely on a remote machine or server.

[0077] In the context of the present application, a machine-readable storage medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable storage medium can be a machine-readable signal storage medium or a machine-readable storage medium. A machine-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0078] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0079] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0080] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. The client-server relationship is established by computer programs running on the respective computers and establishing a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service system that addresses the management difficulties and poor business scalability of traditional physical hosts and VPS services ("Virtual Private Servers," or simply "VPS"). The server may also be a server in a distributed system or a server integrated with blockchain.

[0081] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of this application can be achieved. This is not limited herein.

[0082] The above specific embodiments do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application shall be included in the scope of protection of this application.

Claims

1. A method for migrating data, comprising: Get the identifier of the target data; Check whether the status corresponding to the identifier is pending migration; In response to determining that the state corresponding to the identifier is migrated, obtaining an identifier of new data; In response to determining that the state corresponding to the identifier is to be migrated, marking the identifier; Copying the target data from the source database to the target database; In response to determining that the target data copying is completed, removing the mark; In response to determining that incremental data of the target data is detected in the source database after the copying is completed, copying the incremental data from the source database to the target database; In response to determining that the incremental data copying is complete, the incremental data is assembled with the target data.

2. The method according to claim 1, wherein The method further comprises: The identifier is searched in a preset data status list.

3. The method according to claim 2, wherein: The method further comprises: In response to determining that the target data copying is completed, the data status list is updated.

4. The method according to claim 1, wherein The method further comprises: Generate migration logs during the copy process; After the copying is completed, comparing the data copied to the target database with the target data; In response to determining that the two are different, a rollback operation is performed according to the migration log and the data copied to the target database is deleted.

5. The method according to claim 4, wherein The migration log includes migration batches; as well as The method further comprises: In response to determining that an error occurs in the data in the target database, other data in the same migration batch as the erroneous data is verified.

6. The method according to claim 4, wherein: The method further comprises: In response to determining that the two are the same, determining the duration between the target data migration completion time and the current time; In response to determining that the duration is greater than a preset duration, the target data in the source database is deleted.

7. A device for migrating data, comprising: an acquisition unit configured to acquire an identification of target data; a marking unit configured to verify whether the state corresponding to the identifier is to be migrated, and mark the identifier in response to determining that the state corresponding to the identifier is to be migrated; The acquiring unit is further configured to acquire an identifier of new data in response to determining that the state corresponding to the identifier is migrated; a replication unit, configured to replicate the target data from a source database to a target database; a removing unit configured to remove the mark in response to determining that the target data copying is completed; The assembly unit is configured to: in response to determining that incremental data of the target data is detected after the copying is completed, copy the incremental data to the target database; in response to determining that the incremental data copying is completed, assemble the incremental data with the target data.

8. The device according to claim 7, wherein The marking unit is further configured to: The identifier is searched in a preset data status list.

9. The device according to claim 8, wherein The apparatus further includes an updating unit configured to: In response to determining that the target data copying is completed, the data status list is updated.

10. The device according to claim 7, wherein The device further comprises: a generation unit, configured to generate a migration log during the replication process; a comparing unit configured to compare the data copied to the target database with the target data after the copying is completed; The rollback unit is configured to, in response to determining that the two are different, perform a rollback operation according to the migration log and delete the data copied to the target database.

11. The device according to claim 10, wherein The migration log includes a migration batch; and The device further includes a verification unit configured to: In response to determining that an error occurs in the data in the target database, other data in the same migration batch as the erroneous data is verified.

12. The device according to claim 10, wherein The apparatus further includes a deleting unit configured to: In response to determining that the two are the same, determining the duration between the target data migration completion time and the current time; In response to determining that the duration is greater than a preset duration, the target data in the source database is deleted.

13. An electronic device for executing a method for migrating data, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are configured to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images