A data migration method and related device

Abstract

Embodiments of the present application provide a data migration method and related equipment, which are used to reduce the workload of data migration and improve the reliability of the system when the data is migrated to the database. The method of the embodiments of the present application comprises: acquiring snapshot data of a first database in a first server, and transmitting the snapshot data to a second database of a second server; wherein the snapshot data is used to maintain a data synchronization state between the second server and the first server; recording a transaction completion time point at which a client completes all transactions, acquiring session connection data of the client which is associated with the transaction completion time point; transmitting the session connection data to the second server, and disconnecting the communication connection between the first server and the client, so that the second server establishes a communication connection with the client according to the session connection data.

Description

Technical Field

[0001] This application relates to the field of data processing, and more particularly to a data migration method and related equipment. Background Technology

[0002] When upgrading database server hardware, operating system, or the database itself, the database needs to be migrated. As the central node of most applications, the migration of the database has a huge impact.

[0003] Current solutions often involve creating a slave database of the source database on the target server and then directly switching the target server's IP address to the source server's IP address. Since the target server lacks the context information of existing connection sessions, this inevitably leads to client connection drops, exceptions, and service interruptions. Furthermore, the lack of checks on transaction completion may interrupt ongoing transactions, resulting in data inconsistencies. Therefore, the current approach of stopping all applications, modifying configurations, and restarting is labor-intensive and prone to service interruptions. Summary of the Invention

[0004] This application provides a data migration method and related equipment to reduce the workload of data migration and improve the reliability of the system when migrating data to a database.

[0005] A first aspect of this application provides a data migration method applied to a first server, the first server being communicatively connected to a second server and a client, the method comprising:

[0006] Obtain snapshot data from the first database in the first server, and transmit the snapshot data to the second database in the second server; wherein the snapshot data is used by the second server to maintain data synchronization with the first server;

[0007] Record the transaction completion time of the client when all transactions are completed, and obtain the session connection data associated with the transaction completion time.

[0008] The session connection data is transmitted to the second server, and the communication connection between the first server and the client is disconnected, so that the second server can establish a communication connection with the client based on the snapshot data and the session connection data.

[0009] Optionally, obtaining snapshot data from the first database in the first server and transmitting the snapshot data to the second database in the second server includes:

[0010] Obtain snapshot data from the first database, export the snapshot data to the second database, and record the snapshot time point when the first server exported the snapshot data;

[0011] The snapshot time point is set as the time point for incremental data synchronization between the first server and the second server, so that when the first server obtains incremental data, the second server synchronously obtains the incremental data; wherein, the incremental data is the data obtained by the first server after the snapshot time point.

[0012] Optionally, recording the transaction completion time points when the client completes all transactions and obtaining session connection data associated with the transaction completion time points includes:

[0013] Obtain all communication connections between the clients and the first server and all transactions, and monitor the completion time of all transactions; wherein, all transactions include a first transaction and a second transaction, the first transaction being the first transaction executed by the client among all transactions, and the second transaction being the last transaction executed by the client among all transactions;

[0014] When the first transaction is completed, the communication connection corresponding to the first transaction is paused, and a timer is started to set a preset waiting time;

[0015] When the second transaction is completed, and the transaction completion time of the second transaction is less than the preset waiting time, the transaction completion time is recorded, and all communication connections between the clients and the first server are suspended.

[0016] Save the session connection data of all clients that communicated with the first server based on the transaction completion time.

[0017] Optionally, the method further includes:

[0018] When the transaction completion time is greater than the preset waiting time, the communication connection between all clients and the first server is restored.

[0019] Set a preset delay duration and execute the step of pausing the communication connection corresponding to the first transaction and starting a timer when the first transaction is completed, so as to set a preset waiting time, until the transaction completion time is less than the preset waiting time.

[0020] Optionally, disconnecting the communication connection between the first server and the client includes:

[0021] The IP address of the first server is deleted according to the Internet Protocol IP address deletion instruction, and the IP address of the first server is set on the second server, so that the second server can establish a communication connection with the client.

[0022] A second aspect of this application provides a data migration method applied to a second server, the second server being communicatively connected to a first server, the method comprising:

[0023] The second server receives snapshot data sent by the first database in the first server; wherein the snapshot data is used by the second server to maintain data synchronization with the first server.

[0024] Receive session connection data sent by the first server;

[0025] A communication connection with the client is established based on the session connection data.

[0026] Optionally, receiving snapshot data sent by the first database in the first server includes:

[0027] Receive snapshot data exported from the first database;

[0028] When the first server acquires incremental data after the snapshot time point, it receives the incremental data transmitted by the first database; wherein, the snapshot time point is the time point at which the first server exports the snapshot data, and the incremental data is the data acquired by the first server after the snapshot time point.

[0029] Optionally, receiving session connection data sent by the first server includes:

[0030] The server receives session connection data generated when the first server and the client establish a communication connection at the transaction completion time point; wherein, the transaction completion time point is the completion time point when the client completes all transactions;

[0031] The step of establishing a communication connection with the client based on the session connection data includes:

[0032] Establish a communication address and communication port identical to the first server based on the session connection data;

[0033] A communication connection is established with the client based on the communication address and the communication port, so that the client can transmit communication data after the transaction completion time to the second server based on the transaction completion time.

[0034] A third aspect of this application provides a data migration system applied to a first server, comprising:

[0035] The acquisition unit is used to acquire snapshot data of the first database in the first server and transmit the snapshot data to the second database of the second server;

[0036] The recording unit is used to record the transaction completion time points when the client completes all transactions and to obtain session connection data associated with the transaction completion time points;

[0037] The transmission and disconnection unit is used to transmit the session connection data to the second server and disconnect the communication connection between the first server and the client, so that the second server can establish a communication connection with the client based on the snapshot data and the session connection data.

[0038] The third aspect of this application provides a method for performing the data migration method described in the first aspect.

[0039] A fourth aspect of this application provides a data migration system applied to a second server, comprising:

[0040] A receiving unit is configured to receive snapshot data sent by the first database in the first server;

[0041] The receiving unit is further configured to receive session connection data sent by the first server, wherein the session connection data includes the transaction completion time point when the client completes all transactions;

[0042] The establishment unit is used to establish a communication connection with the client based on the session connection data and the snapshot data, and to establish communication transmission with the client through the transaction completion time point.

[0043] The fourth aspect of this application provides a method for performing the data migration method described in the second aspect.

[0044] A fifth aspect of this application provides a data migration apparatus, comprising:

[0045] Central processing unit, memory, input / output interfaces, wired or wireless network interfaces, and power supply;

[0046] The memory is either a short-term storage memory or a persistent storage memory;

[0047] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform the data migration method described in the first or second aspect.

[0048] A sixth aspect of this application provides a computer-readable storage medium, characterized in that the computer-readable storage medium includes instructions that, when executed on a computer, cause the computer to perform the data migration method described in the first or second aspect.

[0049] As can be seen from the above technical solutions, the embodiments of this application have the following advantages: The data migration method disclosed in this application first obtains snapshot data of the first database in the first server and transmits the snapshot data to the second database of the second server; then, it records the transaction completion time points when the client completes all transactions and obtains session connection data associated with these transaction completion time points; finally, it transmits the session connection data to the second server and disconnects the communication connection between the first server and the client, so that the second server can establish a communication connection with the client based on the session connection data. Thus, the first server can completely transmit the data previously communicated with the client to the second server, and communication between the client and the first server can also be switched to the second server. Furthermore, this data migration method has a transparent process flow, which can minimize the migration workload and improve system reliability. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0051] Figure 1 This is a schematic diagram of the architecture of a data migration system disclosed in an embodiment of this application;

[0052] Figure 2 This is a flowchart illustrating a data migration method applied to a first server, as disclosed in an embodiment of this application.

[0053] Figure 3 This is a flowchart illustrating a data migration method applied to a second server, as disclosed in an embodiment of this application.

[0054] Figure 4 This is an interactive schematic diagram of a data migration method disclosed in an embodiment of this application;

[0055] Figure 5 This is a schematic diagram illustrating the deployment of a migration procedure disclosed in an embodiment of this application;

[0056] Figure 6 This is a schematic diagram illustrating the transaction time of a client executing a transaction, as disclosed in an embodiment of this application.

[0057] Figure 7This is a schematic diagram of the structure of a kernel protocol stack disclosed in an embodiment of this application;

[0058] Figure 8 This is a schematic diagram of the connection of a service protocol stack disclosed in an embodiment of this application;

[0059] Figure 9 This is a schematic diagram of the structure of a data migration system applied to a first server, as disclosed in an embodiment of this application;

[0060] Figure 10 This is a schematic diagram of the structure of a data migration system applied to a second server, as disclosed in an embodiment of this application;

[0061] Figure 11 This is a schematic diagram of the structure of a data migration device disclosed in an embodiment of this application. Detailed Implementation

[0062] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0063] It should be noted that the use of terms such as "first" and "second" in this application is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed in this application.

[0064] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0065] Please see Figure 1 , Figure 1 This is a schematic diagram of the architecture of a data migration system disclosed in an embodiment of this application. It includes a first server 101, a second server 102, and a client 103.

[0066] In the initial communication connection ( Figure 1 In communication 1), the first server 101 is connected to both the second server 102 and the client 103, but the second server 102 is not connected to the client 103. In subsequent communication connections ( Figure 1 In communication 2), the second server 102 is connected to the client 103, while the first server 101 is not connected to the client 103.

[0067] Furthermore, the first server 101 or the second server 102 mentioned above can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud databases, cloud services, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The client 103 can be a smartphone, tablet, laptop, desktop computer, PDA, mobile internet device (MID), wearable device (e.g., smartwatch, smart bracelet), smart computer, smart vehicle, or other smart terminal. Both the first server 101 and the second server 102 can establish a communication connection with the client 103. The communication connection method is not limited; it can be established directly or indirectly through wired communication or wireless communication, depending on the actual application scenario. This application does not impose any restrictions on this. It should be noted that the number of clients 103 is not limited in this embodiment.

[0068] It should be understood that when client 103 is running, it can be compared with the above... Figure 1The first server 101 or the second server 102 shown interacts with each other, enabling the first server 101 or the second server 102 to receive business data (such as UGC videos transmitted by users through client 103) from each client 103. The client 103 can be a social application, instant messaging application, live streaming application, game application, short video application, video application, music application, karaoke application, shopping application, novel application, payment application, or any other client 103 with the function of displaying data information such as text, images, audio, and video. The specific client 103 can be determined according to the actual application scenario requirements and is not limited here. The client 103 can be a standalone client or an embedded sub-client integrated into another client (such as an instant messaging client, social client, etc.), depending on the actual application scenario and is not limited here.

[0069] In conjunction with the above description, it should also be noted that, for ease of understanding, the first server 101 is the source server, which includes the source database. The second server 102 is the target server, which includes the target database. This will be discussed in more detail later.

[0070] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data migration method applied to a first server, as disclosed in an embodiment of this application. It includes steps 201-203.

[0071] 201. Obtain snapshot data from the first database on the first server and transfer the snapshot data to the second database on the second server.

[0072] In this embodiment, the system deploys a migration program on both the first server and the second server, which can complete the entire migration process. Specifically, the migration program can export all data from the first database on the first server and then transfer all data to the second database on the second server.

[0073] In one specific embodiment, the migration procedure can first export all data from the first database as SQL statements, then transmit the SQL statement data to the second server and import it into the second database. Simultaneously, a snapshot method can be used when exporting from the first database, thereby obtaining a snapshot point (i.e., the snapshot time point described above) as the starting point for subsequent increments. This snapshot method does not lock the first database and does not affect its performance. The data exported using the snapshot method can be understood as the snapshot data described above.

[0074] Based on the above embodiments, in another specific embodiment, the first server can also transmit the snapshot time point to the second database via a snapshot method. It should be understood that a snapshot time point is essentially an identifier used to record the database state. The exported snapshot data is the data of the entire database at this point in time. The implementation of snapshot time points may differ in different databases; this application mainly describes MySQL, where the snapshot time point is recorded using the binlog filename and file offset during export.

[0075] Based on the above description, the first database and the second database can maintain synchronization of snapshot data.

[0076] 202. Record the transaction completion time when the client completes all transactions, and obtain the session connection data associated with the transaction completion time.

[0077] Once the snapshot data synchronization between the first and second servers is complete, although the data in the first and second databases is almost identical, the database cannot be directly switched to the second database at this point. This is because: 1) there are still minor differences between the target and source databases; 2) the client may be executing a transaction, and interrupting the transaction could lead to data loss or inconsistency; 3) the network connection's protocol stack buffer may contain some buffered data located between the client and server, which cannot be ignored; and 4) a direct switch would reset the client's connection. Therefore, a suitable time must be found to pause communication on these connections and save the context information.

[0078] Specifically, the first server can obtain all communication connections between clients and itself. For ease of understanding and description, the following description focuses on the interaction process between one client and both the first and second servers. The first server can record the transaction completion times of all client transactions and use this information to obtain the session connection data associated with those transaction completion times. A transaction can be understood as a program that a client needs to execute within a specific timeframe. In this embodiment, the transaction refers to a database transaction, which is an atomic operation. A transaction may contain multiple statements, which either all succeed or all fail (or are not executed). For ease of understanding and description, this will not be elaborated upon further.

[0079] In one specific embodiment, once the first server detects that the client has completed all transactions, it can set the connection status between the client and the first server to a paused state, discard any subsequent data received through this connection, and record the time when the connection was paused, i.e., the transaction completion time. Then, the connection context information is obtained based on this transaction completion time. This context information can be understood as session connection data, i.e., data related to the communication connection at the time all transactions are completed.

[0080] Based on the above embodiments, in another specific embodiment, there may be multiple transactions executed by the client. Correspondingly, the transaction completion time can be understood as the time of the last transaction executed by the client. For ease of understanding and description, this will be described in detail later.

[0081] 203. Transmit the session connection data to the second server and disconnect the communication connection between the first server and the client, so that the second server can establish a communication connection with the client based on the session connection data.

[0082] After the first server obtains the session connection data, it can transmit the session connection data to the second server. Simultaneously, the first server can disconnect from the client. Correspondingly, the second server can then establish a communication connection with the client based on this session connection data. For details on the establishment method, please refer to [link / reference needed]. Figure 3 The description in the text.

[0083] In one specific embodiment, the first server can transmit context information to the second server, thereby enabling the second server to complete a handshake with the client for connection establishment based on the context information. Specifically, [the specific implementation will be discussed later]. Figure 3 and Figure 4 Describe it.

[0084] The data migration method disclosed in this embodiment first obtains snapshot data of a first database in a first server and transmits the snapshot data to a second database in a second server. Next, it records the transaction completion times of all transactions completed by the client and obtains session connection data associated with these transaction completion times. Finally, it transmits the session connection data to the second server and disconnects the communication connection between the first server and the client, allowing the second server to establish a communication connection with the client based on the session connection data. Thus, the first server can completely transmit the data previously communicated with the client to the second server, and communication between the client and the first server can also be switched to the second server. Furthermore, this data migration method is transparent in its process flow, minimizing the migration workload and improving system reliability.

[0085] Please see Figure 3 , Figure 3 This is a flowchart illustrating a data migration method applied to a second server, as disclosed in an embodiment of this application. It includes steps 301-303.

[0086] 301. Receive snapshot data sent by the first database in the first server.

[0087] It is easy to understand that this embodiment mainly describes the process of executing the data migration method on the second server, and its process is similar to... Figure 2 The specifics will not be elaborated here.

[0088] Correspondingly, the second server can receive snapshot data exported from the first database on the first server. Once the second server receives this snapshot data, it can store it in the second database. The snapshot data can be found in [reference needed]. Figure 2 The specific description of snapshot data is not detailed here.

[0089] 302. Receive session connection data sent by the first server.

[0090] Correspondingly, the second server can receive session connection data transmitted by the first server. The session connection data can be found in [reference needed]. Figure 2 The specific description of session connection data is omitted here.

[0091] 303. Establish a communication connection with the client based on the session connection data.

[0092] Once the second server receives the session connection data and snapshot data, it can iterate through the session connection data. Then, using the session connection data and the session connection data obtained by the first server at the transaction completion time, it restores the corresponding communication connection, thereby establishing a communication connection with the client and enabling communication transmission. It's easy to understand that this transaction completion time is the time when the client completes all transactions.

[0093] In one specific embodiment, the second server can retrieve the context information of the communication connection between the first server and the client by traversing the session connection data. Simultaneously, the second server can also obtain the IP address of the first server, and thus construct a communication connection between the server and the client based on that IP address or other context information. Therefore, the second server can possess the same IP address and context information as the first server, and thus set its own IP address to the address of the first server, thereby establishing a communication connection with the client to enable communication transmission between the client and the second server.

[0094] Through the data migration method disclosed in this embodiment, the second server can have the same communication address or communication method as the first server, thereby establishing the same communication connection with the client as the first server. Furthermore, since the series of operations of pausing communication, restoring context, and switching addresses are very short, the database and the client will not perceive any impact, thereby minimizing the loss of data migration.

[0095] For the convenience of understanding the above Figure 2 and Figure 3 The embodiments described herein are described in detail; please refer to [link / reference]. Figure 4 , Figure 4 This is an interactive schematic diagram of a data migration method disclosed in an embodiment of this application. It includes steps 401-409. It should be noted in advance that this embodiment mainly describes the interaction flow between the first server, the second server, and the client. For ease of understanding and description, this will not be elaborated upon further.

[0096] 401. The first server obtains snapshot data from the first database and records the snapshot time point when the first server exported the snapshot data.

[0097] It should be noted in advance that you should refer to [link / reference]. Figure 5 , Figure 5 This is a schematic diagram illustrating the deployment of a migration procedure disclosed in an embodiment of this application. Figure 5 As shown, the system deploys a migration program on both the source database (first server) and the target database (second server). This migration program completes the entire migration process. Some functions of this program reside in the operating system's user space, such as exporting full data from the source database, importing snapshot data into the target database, and enabling master-slave incremental synchronization. Other functions must be executed in the operating system's kernel space, such as capturing and pausing communication traffic from client connections. This is because these tasks require the installation of hook functions in the operating system; on Linux, the netfilter framework is used. The migration programs on the source and target databases communicate to transmit database data, connection session context information, and the timing of certain protocol operations.

[0098] First, the first server can retrieve all data from the current first database. When exporting this data to the second server, it records the time point at which the data is exported. It's easy to understand that this data is the snapshot data described above, and the corresponding time point is the snapshot time point.

[0099] In one specific embodiment, the migration procedure first exports all data from the source database (first database) into SQL statements, then transmits the SQL statement data to the target server (second server) and imports it into the target database (second database). A snapshot method is used when exporting from the source database, thus obtaining a snapshot point as the starting point for subsequent incremental changes. The snapshot method does not lock the source database and does not affect its performance. For MySQL, the `mysqldump` command is used with the `--master-data=2 --single-transaction` parameter. The snapshot time point can be found in [reference needed]. Figure 2 The specific description of this part in step 201 will not be repeated here.

[0100] Based on the above embodiments, it is easy to understand that the data obtained by the first server at this time can be understood as snapshot data.

[0101] 402. The first server sets the snapshot time point to the time point for incremental data synchronization between the first server and the second server.

[0102] Based on step 401, if the first server continues to receive data transmitted by the client, the first server can set the snapshot time point as the time point for synchronizing data between the first server and the second server.

[0103] In one specific embodiment, based on step 401, the source server (first server) can initiate master-slave synchronization from the source database to the target database. Thus, when the first server acquires incremental data (i.e., data acquired after the snapshot time point), the second server can synchronously acquire that incremental data. Specifically, MySQL uses the command `changemaster_host='x', master_port=x, master_user='x', master_password='x', master_log_file='x', master_log_pos=x` to set the master-slave synchronization information. The command sets `master_log_file` and `master_log_pos` to fully export the snapshot information of the binlog log files recorded at that time. This quickly gives the target database almost identical data to the source database.

[0104] 403. The first server obtains all client communication connections and all transactions, and monitors the completion time of all transactions.

[0105] For ease of understanding and description, the following descriptions of the source server, target server, source database, and target database will be referred to as the first server, second server, first database, and second database, respectively.

[0106] Once the first and second servers have completed master-slave synchronization of snapshot and incremental data, although the first and second databases are almost identical, the first database cannot be directly switched to the second database. This is because: 1) there are still minor differences between the second and first databases; 2) the client may be executing a transaction, and interrupting the transaction could lead to data loss or inconsistency; 3) the network connection's protocol stack buffer may contain buffered data between the client and server, which cannot be ignored; and 4) a direct switch would reset the client's connection. Therefore, a suitable time must be found to pause communication on these connections and save the context information. For an understanding of transactions, please refer to [link to relevant documentation]. Figure 2 The description of step 202 is omitted here.

[0107] Therefore, please refer to Figure 7 , Figure 7 This is a schematic diagram of a kernel protocol stack disclosed in an embodiment of this application. In this embodiment, the data migration system needs to monitor and modify network traffic. User-space programs can only obtain data from their own connections. To manipulate the communication data of other processes, hook functions need to be installed in the kernel protocol stack. Therefore, some modules in this system reside in the operating system kernel. In Linux, this is accomplished using the netfilter subsystem, which can capture data from any connection and even modify it. The data capture is used to detect the time point of transaction completion.

[0108] Specifically, the first server can obtain all client communication connections and all transactions, and monitor the completion time of all transactions.

[0109] In one specific embodiment, the first server can obtain all client connections, denoted as db_conn, and ongoing transactions, denoted as db_tx, from the first database. Hook functions are installed in the kernel's netfilter subsystem to intercept database connection communication traffic, parse it according to the MySQL communication protocol, and update db_conn and db_tx in real time, while simultaneously monitoring the completion status of all transactions.

[0110] Specifically, the completion time of all transactions can be monitored. It's easy to understand that since multiple transactions can exist, the logic of all transactions will be described subsequently as "first transaction" and "second transaction." Correspondingly, there may also be a third or fourth transaction, etc., without specific restrictions here. The first transaction is the first transaction executed by the client, and the second transaction is the last transaction executed by the client. Correspondingly, the third or fourth transaction, etc., are transactions executed within the first and second transactions.

[0111] 404. When the first transaction is completed, the first server will pause the communication connection corresponding to the first transaction and start a timer to set a preset waiting time.

[0112] Based on step 403, when the first transaction is completed, the first server can disconnect the connection state corresponding to the first transaction and discard the data subsequently received by this connection. At the same time, a timer is started and a preset waiting time is set to specify and monitor the completion status of the transaction within the preset waiting time.

[0113] In one specific embodiment, when the first server parses and finds that the first transaction has been completed, it records the state of the connection as paused, discards the data subsequently received by the connection, and starts a timer user to record the maximum waiting time.

[0114] It is not difficult to understand that, in another specific embodiment, this can be understood as the first server disconnecting the communication connection with the client corresponding to the first transaction.

[0115] 405. When the transaction completion time is longer than the preset waiting time, the first server restores all client communication connections and sets the preset delay duration.

[0116] Based on step 404, the first server continues to parse the data, waiting for other transactions to complete.

[0117] In one embodiment, when other transactions have not completed all transactions within a predetermined preset waiting time (i.e., the transaction completion time when all transactions are completed is greater than the preset waiting time), the first server restores all communication connections to the client and sets a preset extension time. In other words, the time from the completion of the first transaction to the completion of the second transaction is the transaction completion time.

[0118] In one specific embodiment, the first server continues parsing data. If the timer expires first, it cancels the pause status of all connections and continues receiving data for a period of time t. Here, t is the preset extension time mentioned above. In another specific embodiment, the time t needs to be set as small as possible; no specific restriction is imposed here.

[0119] After setting the preset extension time, continue executing step 404. This continues until the preset extension time is exceeded, or the judgment condition 406 is met.

[0120] 406. When the second transaction is completed, and the completion time of the second transaction is less than the preset waiting time, the first server records the transaction completion time and suspends the communication connection between all clients and the first server.

[0121] Based on step 404, when the second transaction is completed, and the transaction completion time of the second transaction is less than the preset waiting time, the connection is set to a paused state and subsequent received data is discarded.

[0122] In one specific embodiment, while the first server continues to parse data, it waits for other transaction completion events. If a transaction completion event arrives first, the connection is set to a paused state, and subsequently received data is discarded.

[0123] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating the transaction time of a client executing a transaction as disclosed in an embodiment of this application. Figure 6 When executing transactions for each connection, the migration program needs to find a point in time where all transactions are completed. However, without intervention, this point in time may not exist because transactions on different connections might be executed concurrently. Therefore, when a transaction on a particular connection completes, it's necessary to pause data reception on that connection for a period of time, blocking and waiting for all transactions to complete. However, this waiting operation comes at the cost of blocking client operations, so a maximum waiting time is set. If the maximum waiting time is exceeded, data is allowed to pass through all connections for a period before starting the next waiting cycle, until all transactions complete within the maximum waiting time. Regarding the implementation of pausing data reception on connections, since the TCP communication protocol uses sequence numbers to ensure reliable data delivery, it's only necessary not to acknowledge subsequent received data in the kernel hook function; the client will resend this data later. It should also be noted that, from a micro perspective, the database state can be migrated to the new machine by briefly blocking communication with the client. However, the entire migration time is very short, and the implementation will be carried out at a time with low business traffic (such as early morning), so there will be no visible impact on the business.

[0124] 407. The first server saves the session connection data of all clients communicating with the first server based on the transaction completion time.

[0125] When the second transaction is completed, the first server saves the session connection data of all clients when communicating with the first server based on the transaction completion time.

[0126] In one specific implementation, when all transactions for all connections have been committed, the MySQL session context data `db_conn` for all current connections is saved, including session variables, user variables, the current database name, and prepared statements. Simultaneously, the context information `tcp_conn` for all TCP connections is recorded, including source IP, source port, destination IP, destination port, send / receive buffer data, sequence number, TCP options, and sliding window information. It is easy to understand that the data described above is the session connection data. For ease of understanding and description, this will not be elaborated upon further.

[0127] 408. The second server deletes the IP address of the first server according to the IP address deletion command, and sets the Internet Protocol IP address of the first server to the second server.

[0128] The migration procedure for the second server restores the MySQL session context (db_conn) and TCP connection context (tcp_conn) from the first server to the second server. Once the relevant context information has been restored, the IP address of the first server can be deleted using the ipaddrdel command, and the IP address of the second server can be set using the ipaddradd command.

[0129] In one specific embodiment, the second server can iterate through tcp_conn, and for each TCP connection, call the kernel function do_tcp_setsockopt to recover connection information, use the option TCP_REPAIR_QUEUE to recover buffer data, use the option TCP_QUEUE_SEQ to recover sequence numbers, use TCP_REPAIR_OPTIONS to recover TCP options, and use the option TCP_REPAIR_WINDOW to recover sliding window information.

[0130] 409. The second server establishes the same communication address and communication port as the first server based on the session connection data, and establishes a communication connection with the client based on the communication address and communication port, and establishes communication transmission with the client through the transaction completion time point.

[0131] On the second server, the second server can establish the same communication address and communication port as the first server based on the session connection data and snapshot data obtained in the above steps, and forge communication packets based on the communication address and communication port, thereby simulating the client to establish a communication connection with the second server.

[0132] In one specific embodiment, the second server iterates through `tcp_conn` to retrieve the client's IP address and port. It then uses this information to forge a communication connection from the client's IP address and port to the second server's IP address and port. For port-related data packets, the destination MAC address is set to the local machine's MAC address, and the packets are sent directly to the second server's network interface card (NIC) via a raw socket. This method constructs and sends data packets, completing the three-way handshake for connection establishment. It's important to note that the "destination MAC address" refers to the MAC address selection option in the data packet's padding (or options). Selecting "local MAC address" ensures that the MAC address for subsequent migrations is the second server's address; in other words, "local machine" refers to the second server during migration. Simultaneously, the second server iterates through `db_conn` to construct data packets containing SQL statements used to restore the database session context. It executes the statement `setx = a` to restore session variables, `set@x = a` to restore user variables, `usedb` to restore the current database, and `preparestmt fromxxx` to restore prepared statements. These data packets are sent using a raw socket, masquerading as a client, to complete the session restoration.

[0133] It is easy to understand that the above is only one way for the second server to establish a communication connection with the client. For another specific embodiment, please refer to... Figure 8 Figure 8 This is a schematic diagram of a business protocol stack disclosed in an embodiment of this application. Specifically, after all communication between the first database and the client is paused, the migration program transmits the MySQL and TCP connection context information from the first server to the second server. The migration program on the second server will then impersonate all clients connecting to the second server by directly sending underlying data packets, thereby restoring the MySQL database connection on the second server. Subsequently, it reconstructs the context information of each connection's session by constructing MySQL business protocol packets, including session variables, user variables, the current database name, and prepared statements. Finally, the TCP connection information is restored.

[0134] Therefore, after the context information is restored, the second server has the same MySQL session and TCP connection context as the first server. At this point, the IP address of the first server is deleted using the `ipaddrdel` command, and the IP address of the first server is set to the IP address of the second server using the `ipaddradd` command. The servers will then rebind the mapping between IP addresses and MAC addresses using the Address Resolution Protocol (ARP). All data packets will switch from the first server to the second server. Since the first server did not acknowledge data packets after the pause point, and due to the reliable transmission characteristics of the TCP / IP protocol, the client will resend subsequent data packets, so communication will resume from the previous pause point. This series of operations—pausing communication, restoring context, and switching addresses—takes very little time, and the second database and the client will not perceive any impact.

[0135] From the above description, it is easy to understand that this system first exports snapshot data from the source database in snapshot form and records the snapshot points when importing into the target database. Then, it starts master-slave synchronization between the source and target databases and sets it to synchronize from the aforementioned snapshot points. Soon, their data will be almost identical. At the same time, a hook function is installed on the kernel protocol stack of the source database to intercept all communication traffic between the database client and the server. The purpose is to find a suitable point in time to pause communication between the client and the server. At this point, all transactions must have been completed. At this time, the context information of TCP connections and database sessions can be collected. Then, these connections are restored on the target database by masquerading as clients. In this way, the target database has the same connection context information as the source database. By simply changing the IP address of the target server to the IP address of the source server, all client requests will switch to the target server, completing the database migration.

[0136] This application mainly proposes a database migration method that is transparent to the application. The core idea is as follows: a hook function is installed in the operating system kernel to intercept all network traffic, analyze the communication between the client and the database server, and wait for an appropriate time to pause the communication between the client and the server. The context information of each TCP connection and the database session context information are saved at this time, and then resumed on the target server. In this way, the source server and the target server have the same execution context information. Then, the IP address of the source server is bound to the target server. In this way, the communication between the client and the database will switch to the target server, so that the entire migration process is transparent to the database client. The client does not need any configuration modification or restart operation, which ensures the continuity of application system services, reduces the workload of migration, and avoids the operational risks that may be caused by migration.

[0137] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0138] If the plan involves sensitive information (such as user information or corporate information), it should state that the collection, use, and processing of sensitive information must comply with the laws, regulations, and standards of the relevant countries and regions, and must be carried out with the permission or consent of the relevant entities (such as users or enterprises).

[0139] Please see Figure 9 , Figure 9 This is a schematic diagram of the structure of a data migration system applied to a first server, as disclosed in an embodiment of this application.

[0140] The acquisition unit 901 is used to acquire snapshot data of the first database in the first server and transmit the snapshot data to the second database of the second server;

[0141] The recording unit 902 is used to record the transaction completion time point when the client completes all transactions and to obtain session connection data associated with the transaction completion time point; wherein, the snapshot data is used by the second server to maintain data synchronization with the first server;

[0142] The transmission and disconnection unit 903 is used to transmit session connection data to the second server and disconnect the communication connection between the first server and the client, so that the second server can establish a communication connection with the client based on the session connection data.

[0143] For example, the data migration system further includes: a setup unit 904;

[0144] The acquisition unit 901 is specifically used to acquire snapshot data from the first database, export the snapshot data to the second database, and record the snapshot time point when the first server exported the snapshot data.

[0145] Setting unit 904 is used to set the snapshot time point to the time point of incremental synchronization data between the first server and the second server, so that when the first server obtains incremental data, the second server synchronously obtains incremental data; wherein, the incremental data is the data obtained by the first server after the snapshot time point.

[0146] For example, the data migration system includes: an execution unit 905 and a storage unit 906;

[0147] The acquisition unit 901 is specifically used to acquire all communication connections between all clients and the first server and all transactions, and to monitor the completion time of all transactions; wherein, all transactions include the second transaction and the first transaction, the first transaction being the first transaction executed by the client in all transactions, and the second transaction being the last transaction executed by the client in all transactions;

[0148] The execution unit 905 is used to pause the communication connection corresponding to the first transaction and start a timer to set a preset waiting time when the first transaction is completed.

[0149] The recording unit 904 is specifically used to record the transaction completion time when the second transaction is completed and the transaction completion time of the second transaction is less than the preset waiting time, and to suspend the communication connection between all clients and the first server.

[0150] Storage unit 906 is used to save session connection data of all clients communicating with the first server according to the transaction completion time.

[0151] For example, the data migration system further includes: a recovery unit 907;

[0152] Recovery unit 907 is used to restore the communication connection between all clients and the first server when the transaction completion time is greater than the preset waiting time;

[0153] The setting unit 904 is also used to set a preset delay duration and to perform the steps of pausing the communication connection corresponding to the first transaction and starting a timer when the first transaction is completed, so as to set a preset waiting time, until the transaction completion time is less than the preset waiting time.

[0154] For example, a data migration system includes:

[0155] The setting unit 904 is specifically used to delete the IP address of the first server according to the Internet Protocol IP address deletion instruction, and set the IP address of the first server to the second server, so that the second server can establish a communication connection with the client.

[0156] Please see Figure 10 , Figure 10This is a schematic diagram of the structure of a data migration system applied to a second server, as disclosed in an embodiment of this application.

[0157] The receiving unit 1001 is used to receive snapshot data sent by the first database in the first server; wherein the snapshot data is used by the second server to maintain a data synchronization state with the first server.

[0158] The receiving unit 1001 is also used to receive session connection data sent by the first server;

[0159] Establishment unit 1002 is used to establish a communication connection with the client based on session connection data.

[0160] For example, a data migration system includes:

[0161] The receiving unit 1001 is specifically used to receive snapshot data exported from the first database;

[0162] The receiving unit 1001 is further configured to receive incremental data transmitted by the first database when the first server acquires incremental data after the snapshot time point; wherein, the snapshot time point is the time point at which the first server exports the snapshot data, and the incremental data is the data acquired by the first server after the snapshot time point.

[0163] For example, a data migration system includes:

[0164] The receiving unit 1001 is specifically used to receive session connection data generated when the first server and the client establish a communication connection at the transaction completion time point; wherein, the transaction completion time point is the completion time point when the client completes all transactions;

[0165] The data migration system includes:

[0166] Establishment unit 1002 is specifically used to establish the same communication address and communication port as the first server based on the session connection data;

[0167] The establishment unit 1002 is also used to establish a communication connection with the client based on the communication address and communication port, so that the client can transmit communication data after the transaction completion time to the second server based on the transaction completion time.

[0168] Please refer to the following: Figure 11 A schematic diagram of the structure of a data migration device disclosed in this application includes:

[0169] Central processing unit 1101, memory 1105, input / output interface 1104, wired or wireless network interface 1103, and power supply 1102;

[0170] Memory 1105 is either a short-term storage memory or a persistent storage memory;

[0171] The central processing unit 1101 is configured to communicate with the memory 1105 and execute instructions stored in the memory 1105 to perform the aforementioned operations. Figures 2 to 4 The method in any of the embodiments shown.

[0172] This application also provides a chip system, characterized in that the chip system includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected via a circuit, and the at least one processor is used to run computer programs or instructions to perform the aforementioned... Figures 2 to 4 The method in any of the embodiments shown.

[0173] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0174] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0175] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0176] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0177] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data migration method, characterized in that, Applied to a first server, which is communicatively connected to a second server and a client, the method includes: Obtain snapshot data from the first database in the first server, and transmit the snapshot data to the second database in the second server; wherein the snapshot data is used by the second server to maintain data synchronization with the first server; Record the transaction completion time of the client when all transactions are completed, and obtain the session connection data of the client that is associated with the transaction completion time. The session connection data is transmitted to the second server, and the communication connection between the first server and the client is disconnected, so that the second server can establish a communication connection with the client based on the session connection data; The process of recording the transaction completion time points when the client completes all transactions and obtaining session connection data associated with the transaction completion time points includes: Obtain all communication connections between the clients and the first server and all transactions, and monitor the completion time of all transactions; wherein, all transactions include a first transaction and a second transaction, the first transaction being the first transaction executed by the client among all transactions, and the second transaction being the last transaction executed by the client among all transactions; When the first transaction is completed, the communication connection corresponding to the first transaction is paused, and a timer is started to set a preset waiting time; When the second transaction is completed, and the transaction completion time of the second transaction is less than the preset waiting time, the transaction completion time is recorded, and all communication connections between the clients and the first server are suspended. Based on the transaction completion time, save the session connection data of all clients that communicated with the first server.

2. The data migration method according to claim 1, characterized in that, The step of obtaining snapshot data from the first database in the first server and transmitting the snapshot data to the second database in the second server includes: Obtain snapshot data from the first database, export the snapshot data to the second database, and record the snapshot time point when the first server exported the snapshot data; The snapshot time point is set as the time point for incremental data synchronization between the first server and the second server, so that when the first server obtains incremental data, the second server synchronously obtains the incremental data; wherein, the incremental data is the data obtained by the first server after the snapshot time point.

3. The data migration method according to claim 1, characterized in that, The method further includes: When the transaction completion time is greater than the preset waiting time, the communication connection between all clients and the first server is restored. Set a preset delay duration and execute the step of pausing the communication connection corresponding to the first transaction and starting a timer when the first transaction is completed, so as to set a preset waiting time, until the transaction completion time is less than the preset waiting time.

4. The data migration method according to claim 1, characterized in that, Disconnecting the communication connection between the first server and the client includes: The IP address of the first server is deleted according to the Internet Protocol IP address deletion instruction, and the IP address of the first server is set on the second server, so that the second server can establish a communication connection with the client.

5. A data migration method, characterized in that, Applied to a second server, which is communicatively connected to the first server, the method includes: The second server receives snapshot data sent by the first database in the first server; wherein the snapshot data is used by the second server to maintain data synchronization with the first server. Receive session connection data sent by the first server; A communication connection with the client is established based on the session connection data; The step of receiving session connection data sent by the first server includes: The server receives session connection data generated when the first server and the client establish a communication connection at the transaction completion time point; wherein, the transaction completion time point is the completion time point when the client completes all transactions; The step of establishing a communication connection with the client based on the session connection data includes: Establish a communication address and communication port identical to the first server based on the session connection data; A communication connection is established with the client based on the communication address and the communication port, so that the client can transmit communication data after the transaction completion time to the second server based on the transaction completion time.

6. The data migration method according to claim 5, characterized in that, The step of receiving snapshot data sent by the first database in the first server includes: Receive snapshot data exported from the first database; When the first server acquires incremental data after the snapshot time point, it receives the incremental data transmitted by the first database; wherein, the snapshot time point is the time point at which the first server exports the snapshot data, and the incremental data is the data acquired by the first server after the snapshot time point.

7. A data migration system, characterized in that, The system is applied to a first server, which is communicatively connected to a second server and a client. The system includes: The acquisition unit is used to acquire snapshot data of the first database in the first server and transmit the snapshot data to the second database of the second server; wherein the snapshot data is used by the second server to maintain a data synchronization state with the first server. The recording unit is used to record the transaction completion time points when the client completes all transactions and to obtain session connection data associated with the transaction completion time points; The transmission and disconnection unit is used to transmit the session connection data to the second server and disconnect the communication connection between the first server and the client, so that the second server can establish a communication connection with the client based on the session connection data. The data migration system includes: an execution unit and a storage unit; The acquisition unit is specifically used to acquire all communication connections between the clients and the first server and all transactions, and to monitor the completion time of all transactions; wherein, all transactions include a first transaction and a second transaction, the first transaction being the first transaction executed by the client among all transactions, and the second transaction being the last transaction executed by the client among all transactions; The execution unit is used to pause the communication connection corresponding to the first transaction and start a timer to set a preset waiting time when the first transaction is completed. The recording unit is specifically used to record the transaction completion time when the second transaction is completed and the transaction completion time of the second transaction is less than the preset waiting time, and to suspend the communication connection between all clients and the first server. A storage unit is used to save all session connection data between the client and the first server based on the transaction completion time.

8. A data migration system, characterized in that, The system is applied to a second server, which is communicatively connected to the first server, and includes: A receiving unit is configured to receive snapshot data sent by a first database in the first server; wherein the snapshot data is used by the second server to maintain a data synchronization state with the first server. The receiving unit is also used to receive session connection data sent by the first server; The establishment unit is used to establish a communication connection with the client based on the session connection data; The data migration system includes: The receiving unit is specifically configured to receive the session connection data generated when the first server and the client establish a communication connection at the transaction completion time point; wherein, the transaction completion time point is the completion time point when the client completes all transactions; The data migration system includes: The establishment unit is specifically used to establish the same communication address and communication port as the first server based on the session connection data. The establishment unit is further configured to establish a communication connection with the client based on the communication address and the communication port, so that the client can transmit communication data after the transaction completion time to the second server based on the transaction completion time.

Images