Data migration method and apparatus

Abstract

The application discloses a data migration method and device, and relates to the technical field of computers. The method comprises the following steps: switching the read-write address of new data from a data source end to a data destination end; starting a double-end reading operation to read the new data from the data source end and the data destination end; in response to the current data migration state belonging to a first state, writing the new data into the data destination end based on a serialization writing mode; after a preset time length, closing the serialization writing mode, and writing the new data into the data destination end; and migrating the inventory data of the data source end to the data destination end. The method can improve the reliability of data migration.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically to data migration methods and apparatus. Background Technology

[0002] Cloud computing is a computing service, and as cloud storage service providers, their storage systems are constantly being developed, upgraded, or replaced. The reliability of data read / write services is crucial when switching between old and new storage systems. Current data migration methods involve migrating existing data from the old storage system to the new one, and mirroring read / write operations on newly generated data from the new storage system back to the old storage system to ensure continuity of data read / write services during the switchover process.

[0003] However, existing data migration methods suffer from poor reliability. Summary of the Invention

[0004] This disclosure provides a data migration method, apparatus, electronic device, and computer-readable storage medium.

[0005] According to a first aspect of this disclosure, a data migration method is provided, comprising: switching the read / write address of new data from the data source end to the data destination end, wherein the new data refers to data generated by the data generation end during the data migration process; enabling a dual-end read operation to read the new data from both the data source end and the data destination end; responding to the current data migration state being a first state, writing the new data to the data destination end based on a serialized write method; disabling the serialized write method after a preset time and writing the new data to the data destination end; and migrating the existing data from the data source end to the data destination end.

[0006] In some embodiments, reading new data from both the data source and the data destination includes: prioritizing reading new data from the data destination; and reading new data from the data source in response to no new data being read from the data destination.

[0007] In some embodiments, in response to the current data migration state being a first state, writing new data to the data destination based on a serialized writing method includes: in response to determining that a first lock exists, determining whether the first lock is valid, wherein the first lock is used to lock data write permissions; in response to determining that the first lock is valid, writing new data to the data destination based on the first lock using a serialized writing method.

[0008] In some embodiments, the data migration method further includes: in response to the current data migration state being a second state, writing new data to the data source based on a serialized writing method.

[0009] In some embodiments, in response to the current data migration state being a second state, writing new data to the data source based on a serialized write method includes: in response to determining that a second lock exists, determining whether the second lock is valid, wherein the second lock is used to lock data write permissions; in response to determining that the second lock is valid, writing new data to the data source based on a first lock using a serialized write method.

[0010] In some embodiments, before initiating the dual-end read operation, the method further includes: verifying the switching operation that switches the read / write address of new data from the data source to the data destination; and in response to confirming that the verification is successful, allocating a backend storage cluster for the switching operation, wherein the backend storage cluster includes the data source or the data destination.

[0011] In some embodiments, the data migration method further includes: disabling the dual-end read operation and reading new data from the data destination.

[0012] According to a second aspect of this disclosure, a data migration apparatus is provided, comprising: a switching unit configured to switch the read / write address of new data from a data source to a data destination, wherein the new data refers to data generated by a data generator during the data migration process; a dual-end reading unit configured to enable dual-end reading operations to read new data from both the data source and the data destination; a first writing unit configured to write the new data to the data destination using a serialized writing method in response to the current data migration state being in a first state; a second writing unit configured to disable the serialized writing method after a preset time and write the new data to the data destination; and a data migration unit configured to migrate existing data from the data source to the data destination.

[0013] In some embodiments, the dual-end reading unit includes: a first reading module configured to preferentially read new data from the data destination end; and a second reading module configured to read new data from the data source end in response to the failure to read new data from the data destination end.

[0014] In some embodiments, the first writing unit includes: a first determining module configured to determine whether the first lock is valid in response to determining the existence of a first lock, wherein the first lock is used to lock data writing permissions; and a first writing module configured to write new data to the data destination based on the first lock in response to determining that the first lock is valid.

[0015] In some embodiments, the data migration apparatus further includes a third writing unit configured to write new data to the data source in response to the current data migration state being in the second state, based on a serialized writing method.

[0016] In some embodiments, the third writing unit includes: a second determining module configured to determine whether the second lock is valid in response to determining the existence of a second lock, wherein the second lock is used to lock data writing permissions; and a second writing module configured to write new data to the data source end based on the serialization writing method in response to determining that the second lock is valid.

[0017] In some embodiments, before initiating the dual-end read operation, the apparatus further includes: verifying a switching operation that switches the read / write address of new data from the data source to the data destination; and, in response to determining that the verification is successful, allocating a backend storage cluster for the switching operation, wherein the backend storage cluster includes either the data source or the data destination.

[0018] In some embodiments, the data migration apparatus further includes a shutdown unit configured to disable the dual-end read operation and read new data from the data destination.

[0019] According to a third aspect of this disclosure, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a data migration method as provided in the first aspect or a method for predicting information as provided in the second aspect.

[0020] According to a fourth aspect of this disclosure, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements a data migration method as provided in the first aspect or a method for predicting information as provided in the second aspect.

[0021] The data migration method and apparatus disclosed herein include: switching the read / write address of new data from the data source to the data destination; enabling dual-end read operation to read new data from both the data source and the data destination; responding to the current data migration state being in the first state, writing the new data to the data destination using a serialized write method; disabling the serialized write method after a preset time and writing the new data to the data destination; and migrating existing data from the data source to the data destination. This solution can be applied to large-scale data hot migration scenarios between heterogeneous storage systems, avoiding long-tail problems in the data migration cycle and data consistency issues, and improving the availability, performance, and reliability of data migration services.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0023] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:

[0024] Figure 1 This is an exemplary system architecture diagram in which embodiments of this application can be applied;

[0025] Figure 2 This is a flowchart of an embodiment of the data migration method according to this application;

[0026] Figure 3 This is a flowchart of another embodiment of the data migration method according to this application;

[0027] Figure 4 This is a flowchart of an application scenario of the data migration method according to this application;

[0028] Figure 5 This is another flowchart of an application scenario in the data migration method of this application;

[0029] Figure 6 This is a schematic diagram of the data writing process in an application scenario of the data migration method according to this application;

[0030] Figure 7 This is a schematic diagram of a structure of an embodiment of the data migration apparatus according to this application;

[0031] Figure 8 This is a block diagram of an electronic device used to implement the data migration method of the embodiments of this application. Detailed Implementation

[0032] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0033] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the data migration method or data migration apparatus of this application can be applied.

[0034] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0035] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be user terminal devices, on which various client applications can be installed, such as data migration software, data maintenance applications, image applications, video applications, search applications, financial applications, etc.

[0036] Terminal devices 101, 102, and 103 can be various electronic devices with displays that support receiving messages from the server, including but not limited to smartphones, tablets, e-book readers, electronic players, laptops, and desktop computers.

[0037] Terminal devices 101, 102, and 103 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules (e.g., multiple software modules used to provide distributed services) or as a single software program or software module. No specific limitations are made here.

[0038] Server 105 can switch the read / write address of new data from the data source to the data destination. Here, new data refers to data generated by the data generator during the data migration process. It can enable dual-end read operation to read new data from both the data source and the data destination. In response to the current data migration status being in the first state, it can write the new data to the data destination using a serialized write method. After a preset time, it can disable the serialized write method and write the new data to the data destination. It can also migrate the existing data from the data source to the data destination.

[0039] It should be noted that the data migration method provided in the embodiments of this disclosure can be executed by the server 105, and correspondingly, the data migration device can be set in the server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0041] Continue to refer to Figure 2 The flowchart 200, illustrating an embodiment of the data migration method according to this disclosure, includes the following steps:

[0042] Step 201: Switch the read / write address of the new data from the data source to the data destination. Here, the new data refers to the data generated by the data generation end during the data migration process.

[0043] In this embodiment, the execution entity of the data migration method (e.g. Figure 1 The server shown can switch the read / write address of new data from the data source to the data destination. Here, "new data" refers to data newly generated at the data generation end during the storage switching process / stage, and to be written to the data storage end. The data source end refers to the data storage end before data migration, and the data destination end refers to the new data storage end to which the data is to be migrated.

[0044] Step 202: Enable dual-end read operation to read new data from both the data source and the data destination.

[0045] In this embodiment, dual-end read operation can be enabled to read new data from both the data source and the data destination during data hot migration. This step does not change the data writing process / operation, but only updates the data reading process to dual read.

[0046] Step 203: In response to the current data migration state being in the first state, write the new data to the data destination using a serialized writing method.

[0047] In this embodiment, if the current data migration status is in the first stage of the storage-side switching phase (i.e., switching the read / write address of the new data from the data source to the data destination), the new data can be written to the data destination using a serialized write method.

[0048] Step 204: After a preset time, the serialization writing mode is turned off, and the new data is written to the data destination.

[0049] In this embodiment, after switching the data writing operation to a preset duration at the data destination, the serialized writing mode of the data destination can be turned off, and the new data can be written to the data destination in the system's default data writing mode or a conventional mode (such as parallel writing mode).

[0050] Step 205: Migrate the existing data from the data source to the data destination.

[0051] In this embodiment, after switching all read and write operations of new data from the data source to the data destination, the existing data at the data source can be migrated to the data destination to complete the migration of all data (existing data and new / incremental data).

[0052] The data migration method provided in this embodiment switches the read / write address of new data from the data source to the data destination; enables dual-end read operation to read new data from both the data source and the data destination; responds to the current data migration state being in the first state, writes the new data to the data destination using a serialized write method; after a preset time, disables the serialized write method and writes the new data to the data destination; and migrates the existing data from the data source to the data destination. This solution can be applied to large-scale hot data migration scenarios between heterogeneous storage systems, avoiding long-tail problems in the data migration cycle and data consistency issues, and improving the availability, performance, and reliability of data migration services.

[0053] Optionally, new data can be read from both the data source and the data destination, including: prioritizing reading new data from the data destination; and reading new data from the data source in response to the absence of new data from the data destination.

[0054] In this embodiment, during a dual-end read operation, new data can be read first from the data destination. If no new data is found at the data destination, then new data is read from the data source. Because the LastModified time of data in a heterogeneous storage system is unreliable—for example, a user might upload data to the destination cluster, but the destination cluster's system time might lag behind the source cluster for some reason—using the last update time to determine whether to return data to the source or destination could lead to unreliable data being unread after an update. Therefore, prioritizing reading new data from the data destination ensures data read consistency.

[0055] Optionally, in response to the current data migration state being in the first state, writing new data to the data destination based on a serialized writing method includes: in response to determining that a first lock exists, determining whether the first lock is valid, wherein the first lock is used to lock data write permissions; in response to determining that the first lock is valid, writing new data to the data destination based on the first lock using a serialized writing method.

[0056] In this embodiment, if it is determined that a first lock exists, it is determined whether the first lock is valid. If the first lock is determined to be valid (e.g., the first lock lease is within the validity period or the first lock is in an active state), the new data is written to the data destination based on the serialization writing method.

[0057] This embodiment uses state locks to perform data writing operations, which can improve system security and facilitate the maintenance of system operation order.

[0058] Continue to refer to Figure 3 The diagram illustrates a flow 300 of another embodiment of the data migration method according to the present disclosure, including the following steps:

[0059] Step 301: Switch the read / write address of the new data from the data source to the data destination.

[0060] Step 302: Start the dual-end read operation to read new data from both the data source and the data destination. Here, new data refers to the data generated by the data generation end during the data migration process.

[0061] Step 303: In response to the current data migration status being in the second state, write the new data to the data source using a serialized writing method.

[0062] In this embodiment, if the current data migration status is in the second state of the storage-side switching phase (i.e., switching the read / write address of new data from the data source to the data destination), the new data is written to the data source using a serialized write method. The first state and the second state belong to different data storage switching states, and the switching state can be in the first or second state based on the instructions of the data migration technician.

[0063] Step 304: In response to the current data migration state being in the first state, write the new data to the data destination using a serialized writing method.

[0064] Step 305: After a preset time, the serialization writing mode is turned off, and the new data is written to the data destination.

[0065] Step 306: Migrate the existing data from the data source to the data destination.

[0066] In this embodiment, the descriptions of steps 301, 302, 304, 305, and 306 are consistent with the descriptions of steps 201, 202, 203, 204, and 205, and will not be repeated here.

[0067] The data migration method provided in this embodiment is compared to... Figure 2 The described embodiments employ different write endpoints at different stages of data migration to perform data migration operations in stages, which can improve the reliability of data migration services.

[0068] Optionally, in response to the current data migration state being in the second state, new data is written to the data source using a serialized writing method, including: in response to determining that a second lock exists, determining whether the second lock is valid, wherein the second lock is used to lock data write permissions; in response to determining that the second lock is valid, writing new data to the data source using a serialized writing method based on the first lock.

[0069] In this embodiment, if a second lock is determined to exist, it is determined whether the second lock is valid. If the second lock is determined to be valid (e.g., the second lock lease is within its validity period or the second lock is in an active state), the new data is written to the data source based on the serialization writing method.

[0070] This embodiment uses state locks to perform data writing operations, which can improve system security and facilitate the maintenance of system operation order.

[0071] In the above combination Figure 2 and Figure 3 In some optional implementations of the described embodiments, before initiating the dual-end read operation, the data migration method further includes: verifying the switching operation that switches the read / write address of the new data from the data source to the data destination; and in response to determining that the verification is successful, allocating a backend storage cluster for the switching operation, wherein the backend storage cluster includes the data source or the data destination.

[0072] In this embodiment, before initiating the dual-end read operation, the data migration method further includes verifying the switching operation that changes the read / write address of the new data from the data source to the data destination, to verify whether the data migration operation / request is legal or authorized. If the verification is successful, a backend storage cluster is allocated for the switching operation.

[0073] In the above combination Figure 2 and Figure 3 In some optional implementations of the described embodiments, the data migration method further includes: disabling the two-end read operation and reading new data from the data destination.

[0074] In this embodiment, after migrating the write operation of new data to the data destination, the dual-end read operation can be turned off, and new data can be read only from the data destination.

[0075] In some application scenarios, data migration can be achieved based on a proxy service. The proxy service can reside below the gateway of the source storage cluster (data source), at the same level as the source and destination storage clusters (data destination). During data migration, by adjusting the gateway's forwarding rules, user traffic is diverted to the proxy service. After completing request preprocessing, the proxy service dynamically schedules read and write traffic according to the migration stage.

[0076] like Figure 4As shown, the migration proxy service mainly consists of a preprocessing module, a protocol adaptation module, and a routing and forwarding module. The preprocessing module initializes the request context, authenticates (authorizes) data migration requests, and outputs audit logs. The protocol adaptation module dispatches requests to the corresponding API (Application Programming Interface) of the proxy layer. The routing and forwarding module dynamically routes different migration tasks to the appropriate backend storage clusters (data source / data destination) based on the current migration progress.

[0077] like Figure 5 As shown, the data migration process implemented by the proxy service consists of the following stages. During the migration, the implementers dynamically adjust the user's migration status in the proxy service according to the user's migration progress, thereby routing read and write traffic. The entire migration process ensures both the availability of the storage service and the consistency of the data, and the user is unaware of the entire migration process.

[0078] Reading and writing to the data source: When providing proxy access to users, it is necessary to modify the forwarding rules of the source cluster gateway. For example, taking Nginx (a high-performance Hypertext Transfer Protocol and reverse proxy web server) as an example, changing the forwarding rules requires modifying the Nginx configuration file, adjusting the upstream (backend server) property, and executing the Nginx reload command. Since Nginx may be deployed in multiple instances, some user traffic may be routed to the proxy service, while some traffic may remain in the source storage cluster. Therefore, traffic routed to the proxy service is first configured to route both reads and writes to the source storage cluster, enabling the proxy service to achieve a layer of transparent transmission and avoiding risks to data consistency.

[0079] Dual-end data reading and writing to the data source: After the source cluster gateway completes the traffic redirection configuration for the user, it changes the user's traffic status to dual-end data reading and writing to the data source. This step does not change the destination of the written data; it only changes the read data to dual reads to avoid the linear consistency problem of not being able to read data after writing. In the dual-end data reading operation, the destination cluster is read first. If the data exists, it is returned; otherwise, the source cluster is read and the final result (data exists or data not found) is returned directly. Because the last modified time (LastModified) of data in heterogeneous storage systems is unreliable—for example, a user uploads data to the destination cluster, but the system time of the destination cluster may lag behind the source cluster for some reason—if the last update time of the data is used to determine which side to return data, the reliability problem of not being able to read data after updating will occur. Therefore, the destination cluster is read first, rather than reading from both ends simultaneously.

[0080] Dual-end data reading and serialized write to the data source: Enable serialized write, such as... Figure 6 As shown, a resource lock is acquired before writing data. The first lock is a priority lock. When the data migration phase is in the stage of "reading data from both ends and writing to the data source using a serialized writing method," it can be called the left lock (second lock); when the data migration phase is in the stage of "reading data from both ends and writing to the data destination using a serialized writing method," it is called the right lock (first lock), and the left lock has a lower priority than the right lock. The specific steps are as follows:

[0081] If the data migration phase involves "reading data from both ends and writing to the data source using a serialized write method," then a left lock is acquired for the resource to be written. If no lock exists, or if a left lock exists but its lease has expired, then the left lock is acquired and the data is written to the source. If the current left lock lease has not expired or a right lock exists, then this request fails.

[0082] If the data migration phase is in the "dual-end data reading and serialized write to the destination" stage, then acquire the right lock of the resource to be written. If there is no unreleased right lock, then acquire the lock and write the data to the destination.

[0083] Dual-end data reading and writing destination: After the data writing switch is completed, wait for all proxy service instances to be ready after a preset time (some data requests in progress have not yet been completed during the switch). Serialization can be turned off at this time. It is safe to turn off serialization writing at this time because all nodes are writing to the destination cluster, and the linear consistency of data within a single system can be guaranteed.

[0084] After the above steps are completed, the user's incremental writes (new data generated during the migration process) have been safely migrated to the destination cluster. At this point, the migration of existing data can be carried out. Since the data in the destination cluster can be determined to be up-to-date, only the data that does not exist in the destination cluster needs to be migrated.

[0085] After the user's existing data and incremental writes have been switched to the target cluster, dual reads can be turned off to switch both reads and writes to the target cluster. At this point, the entire data migration process is complete, and the proxy service has completed its task.

[0086] This method can be applied to large-scale data hot migration scenarios between heterogeneous storage systems, solving the long tail problem of migration cycle and data consistency problem, and ensuring service availability, performance and data reliability.

[0087] Further reference Figure 7 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a data migration apparatus, which is similar to... Figure 2 and Figure 3Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0088] like Figure 7 As shown, the data migration device of this embodiment includes: a switching unit 701, a dual-end reading unit 702, a first writing unit 703, a second writing unit 704, and a data migration unit 705. The switching unit is configured to switch the read / write address of new data from the data source end to the data destination end, where the new data refers to data generated by the data generation end during the data migration process. The dual-end reading unit is configured to enable dual-end reading operations to read new data from both the data source end and the data destination end. The first writing unit is configured to write the new data to the data destination end using a serialized writing method in response to the current data migration state being in a first state. The second writing unit is configured to disable the serialized writing method after a preset time and write the new data to the data destination end. The data migration unit is configured to migrate existing data from the data source end to the data destination end.

[0089] In some embodiments, the dual-end reading unit includes: a first reading module configured to preferentially read new data from the data destination end; and a second reading module configured to read new data from the data source end in response to the failure to read new data from the data destination end.

[0090] In some embodiments, the first writing unit includes: a first determining module configured to determine whether the first lock is valid in response to determining the existence of a first lock, wherein the first lock is used to lock data writing permissions; and a first writing module configured to write new data to the data destination based on the first lock in response to determining that the first lock is valid.

[0091] In some embodiments, the data migration apparatus further includes a third writing unit configured to write new data to the data source in response to the current data migration state being in the second state, based on a serialized writing method.

[0092] In some embodiments, the third writing unit includes: a second determining module configured to determine whether the second lock is valid in response to determining the existence of a second lock, wherein the second lock is used to lock data writing permissions; and a second writing module configured to write new data to the data source end based on the serialization writing method in response to determining that the second lock is valid.

[0093] In some embodiments, before initiating the dual-end read operation, the apparatus further includes: verifying a switching operation that switches the read / write address of new data from the data source to the data destination; and, in response to determining that the verification is successful, allocating a backend storage cluster for the switching operation, wherein the backend storage cluster includes either the data source or the data destination.

[0094] In some embodiments, the data migration apparatus further includes a shutdown unit configured to disable the dual-end read operation and read new data from the data destination.

[0095] Each unit in the aforementioned device 700 and the reference Figure 2 , Figure 3 The steps described in the method correspond to those steps. Therefore, the operations, features, and technical effects described above for the data migration method also apply to the device 700 and the units contained therein, and will not be repeated here.

[0096] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0097] like Figure 8 The diagram shown is a block diagram of an electronic device 800 for a data migration method according to an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0098] like Figure 8 As shown, the electronic device includes one or more processors 801, a memory 802, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take the 801 processor as an example.

[0099] The memory 802 is the non-transitory computer-readable storage medium provided in this application. This memory stores instructions executable by at least one processor to cause the at least one processor to perform the data migration method provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the data migration method provided in this application.

[0100] Memory 802, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the data migration method in the embodiments of this application (e.g., appendix). Figure 7 The switching unit 701, dual-end reading unit 702, first writing unit 703, second writing unit 704, and data migration unit 705 are shown. The processor 801 executes various server functions and data processing by running non-transient software programs, instructions, and modules stored in the memory 802, thereby implementing the data migration method in the above method embodiment.

[0101] Memory 802 may include a program storage area and a data storage area. The program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device for extracting video clips. Furthermore, memory 802 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 802 may optionally include memory remotely located relative to processor 801, and this remote memory may be connected to the electronic device for extracting video clips via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0102] The electronic device for the data migration method may further include: an input device 803, an output device 804, and a bus 805. The processor 801, memory 802, input device 803, and output device 804 can be connected via the bus 805 or other means. Figure 8 Taking the connection between China and Israel via bus 805 as an example.

[0103] Input device 803 can receive input digital or character information, and generate key signal inputs related to user settings and function control of the electronic device used to extract video clips, such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, joysticks, etc. Output device 804 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touch screen.

[0104] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0105] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0106] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0107] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0108] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0109] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0110] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data migration method based on a proxy service, wherein the proxy service is located below the gateway at the data source end and is at the same level as the data source end and the data destination end, the method comprising: The read / write address of the new data is switched from the data source to the data destination. The new data refers to the data generated by the data generation end during the data migration process. By adjusting the forwarding rules of the gateway, the user's traffic is diverted to the proxy service. After the proxy service completes the request preprocessing, it dynamically schedules the read / write traffic according to the migration stage. The preprocessing includes: request context initialization, identification of data migration requests, and output of audit logs. Enable dual-end read operation to read the new data from both the data source and the data destination. In response to the current data migration status being in the first state of the storage switching phase, the new data is written to the data destination using a serialized writing method; After a preset time, the serialization writing mode is turned off, and the new data is written to the data destination in either the default mode or the parallel mode. After switching all read and write operations for new data from the data source to the data destination, the existing data from the data source is migrated to the data destination.

2. The method according to claim 1, wherein, The step of reading the new data from the data source and the data destination includes: The new data is read first from the data destination. In response to the failure to read the new data from the data destination, the new data is read from the data source.

3. The method according to claim 1, wherein, The step of responding to the current data migration state being in the first state by writing the new data to the data destination terminal based on a serialized writing method includes: In response to determining that a first lock exists, it is determined whether the first lock is valid, wherein the first lock is used to lock data write permissions; In response to determining that the first lock is valid, the new data is written to the data destination in a serialized write mode based on the first lock.

4. The method according to claim 1, wherein, The method further includes: In response to the current data migration status being in the second state, the new data is written to the data source using a serialized writing method.

5. The method according to claim 4, wherein, The response that the current data migration state belongs to the second state includes writing the new data to the data source end based on a serialized writing method, including: In response to determining that a second lock exists, it is determined whether the second lock is valid, wherein the second lock is used to lock data write permissions; In response to determining that the second lock is valid, the new data is written to the data source in a serialized write mode based on the first lock.

6. The method according to any one of claims 1-5, wherein, Before enabling dual-end read operations, the method further includes: Verify the switching operation that changes the read / write address of new data from the data source to the data destination; In response to the successful verification, a backend storage cluster is allocated for the switching operation, wherein the backend storage cluster includes a data source or a data destination.

7. The method according to any one of claims 1-5, wherein, After migrating the existing data from the data source to the data destination, the method further includes: Disable dual-end read operation and read the new data from the data destination end.

8. A data migration apparatus based on a proxy service, wherein the proxy service is located below the gateway at the data source end and is at the same level as the data source end and the data destination end, the apparatus comprising: The switching unit is configured to switch the read / write address of new data from the data source to the data destination. The new data refers to the data generated by the data generation end during the data migration process. By adjusting the gateway's forwarding rules, the user's traffic is diverted to the proxy service. After the proxy service completes request preprocessing, it dynamically schedules read / write traffic according to the migration stage. The preprocessing includes: request context initialization, identification of data migration requests, and output of audit logs. The dual-end reading unit is configured to enable dual-end reading operation to read the new data from both the data source end and the data destination end; The first writing unit is configured to write the new data to the data destination terminal based on a serialized writing method in response to the first state where the current data migration state is in the storage end switching phase. The second writing unit is configured to disable the serialization writing mode after a preset time and write the new data to the data destination in either the default mode or the parallel mode. The data migration unit is configured to migrate the existing data from the data source to the data destination after switching all read and write operations of new data from the data source to the data destination.

9. The apparatus according to claim 8, wherein, The dual-end reading unit includes: The first reading module is configured to prioritize reading the new data from the data destination. The second reading module is configured to read the new data from the data source in response to the failure to read the new data from the data destination.

10. The apparatus according to claim 8, wherein, The first writing unit includes: The first judgment module is configured to determine whether the first lock is valid in response to the determination that a first lock exists, wherein the first lock is used to lock data write permissions; The first write module is configured to, in response to determining that the first lock is valid, write the new data to the data destination in a serialized write mode based on the first lock.

11. The apparatus according to claim 8, wherein, The device further includes: The third writing unit is configured to write the new data to the data source in response to the current data migration state being in the second state, based on a serialized writing method.

12. The apparatus according to claim 11, wherein, The third writing unit includes: The second judgment module is configured to determine whether the second lock is valid in response to the determination that a second lock exists, wherein the second lock is used to lock data write permissions; The second write module is configured to write the new data to the data source in a serialized write mode based on the first lock in response to determining that the second lock is valid.

13. The apparatus according to any one of claims 8-12, wherein, Before enabling dual-end reading operation, the device further includes: Verify the switching operation that changes the read / write address of new data from the data source to the data destination; In response to the successful verification, a backend storage cluster is allocated for the switching operation, wherein the backend storage cluster includes a data source or a data destination.

14. The apparatus according to any one of claims 8-12, wherein, The device further includes: The closing unit is configured to disable the dual-end read operation and read the new data from the data destination end.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

Images