Redis database migration system, method, and electronic device

Abstract

The application provides a Redis database migration system, which comprises a processor and a memory connected in communication, and a configuration file is stored in the memory; the processor is connected in communication with one or more display devices, is used for executing a computer program, displays a configuration page on the display device based on the configuration file, and the configuration page is used for configuring migration information, wherein the migration information at least comprises an ID of a database, environment information, a connection port, data that needs to be migrated in a source database, threads that are needed for migration, a migration tool, a detection tool, a detection frequency and detection accuracy; and the processor is further used for executing a corresponding migration operation according to the obtained migration information. The application also provides a Redis database migration method and an electronic device. The application can save time, improve migration efficiency, and automatically detect migration effects after database migration, so that migration accuracy can be ensured.

Description

Technical Field

[0001] This invention relates to the field of data migration technology, and in particular to a Redis database migration system, method, and electronic device. Background Technology

[0002] Redis is an open-source, high-performance key-value database. It adapts to different storage needs by providing various key-value data types and leverages many high-level interfaces to fulfill various roles such as caching and queuing systems. Redis databases are widely used in diverse business scenarios, resulting in diverse environment configurations. During business operations, data migration within Redis databases often needs to be performed internally or externally for various reasons. Different migration requirements and database configurations necessitate the selection of different data migration tools and command algorithms. However, current technologies often require technicians to individually inspect the database type and configuration, design migration plans for each, and install corresponding software tools to execute the data migration operations. This process is costly in terms of manpower, resources, and time, resulting in low efficiency and a high probability of failure. Furthermore, there is insufficient post-migration evaluation of the migration effect, and errors during the migration process require manual inspection and handling. Currently, however, there is no efficient Redis database migration system applicable to multiple database types. Summary of the Invention

[0003] To address the aforementioned technical problems, the technical solution adopted by this invention is as follows:

[0004] This invention provides a Redis database migration system, comprising: a processor and a memory connected in communication, wherein the memory stores a configuration file; the processor is connected in communication with one or more display devices and is used to execute a computer program to display a configuration page on the display device based on the configuration file; the configuration page is used for users to configure migration information, wherein the migration information includes at least the database ID, environment information, connection port, data to be migrated from the source database, threads required for migration, migration tools, detection tools, number of detections, and detection accuracy;

[0005] The processor is also configured to execute computer programs to perform the following steps:

[0006] S100: Based on the obtained source database and target database IDs and connection ports, select the corresponding docking ports to establish communication connections with the corresponding source database and target database respectively;

[0007] S200: Based on the obtained number of threads required for migration, allocate the corresponding threads for the current migration task, allocate the amount of data to be migrated to each thread based on the obtained data to be migrated, and select the corresponding migration method for the current migration task based on the obtained environment information.

[0008] S300 retrieves the corresponding data from the source database based on the acquired data that needs to be migrated, performs backup, obtains the backup data, and allocates it to the corresponding thread;

[0009] S400: For each thread, the migration tool corresponding to the current migration task is used to migrate the corresponding data to the target database based on the selected migration method. During the migration process, the changed data in the source database is acquired in real time and stored.

[0010] S500: After detecting that the backup data migration is complete, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, and migrate it to the target database; execute S600;

[0011] S600: If no data needs to be migrated in the source database, proceed to S700; otherwise, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, migrate it to the target database, and proceed to S600.

[0012] S700: Based on the number of detections and the detection accuracy, use the detection tool corresponding to the current migration task to check the consistency between the data to be migrated in the source database and the data to be migrated into the target database. If they are consistent, exit the control program; otherwise, execute S800.

[0013] S800: Retrieve inconsistent data from the source database, assign it to the corresponding thread, migrate it to the target database, and exit the control program after migration.

[0014] Another embodiment of the present invention provides a Redis database migration method, the method comprising:

[0015] S10, Obtain migration information, which includes at least the database ID, environment information, connection port, data to be migrated from the source database, the threads required for migration, migration tools, detection tools, number of detections, and detection accuracy;

[0016] S20, Perform the corresponding migration operation based on the acquired migration information;

[0017] S20 specifically includes:

[0018] S201, Based on the obtained source database and target database IDs and connection ports, select the corresponding docking ports to establish communication connections with the corresponding source database and target database respectively;

[0019] S202, based on the obtained number of threads required for migration, allocate the corresponding threads to the current migration task, based on the obtained data to be migrated, allocate the amount of data to be migrated to each thread, and select the corresponding migration method for the current migration task based on the obtained environment information;

[0020] S203: Based on the acquired data that needs to be migrated, retrieve the corresponding data from the corresponding source database for backup, obtain the backup data, and allocate it to the corresponding thread;

[0021] S204, For each thread, the migration tool corresponding to the current migration task is used to migrate the corresponding data to the target database based on the selected migration method. During the migration process, the changed data in the source database is acquired in real time and stored.

[0022] S205, after detecting that the backup data migration is complete, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, and migrate it to the target database; execute S206;

[0023] S206, if no data needs to be migrated in the source database, execute S207; otherwise, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, migrate it to the target database, and execute S206.

[0024] S207: Based on the obtained number of detections and detection accuracy, use the detection tool corresponding to the current migration task to detect the consistency between the data to be migrated in the source database and the data to be migrated into the target database. If they are consistent, exit the control program; otherwise, execute S208.

[0025] S208: Obtain inconsistent data from the source database, assign it to the corresponding thread, migrate it to the target database, and exit the control program after migration.

[0026] Another embodiment of the present invention provides an electronic device, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the processor is used to execute the at least one instruction or the at least one program to implement the aforementioned method.

[0027] The present invention has at least the following beneficial effects:

[0028] This invention can automatically detect the type and configuration of a Redis database during migration, automatically design a migration plan based on migration requirements, and automatically load the corresponding software tools to perform the data migration operation. This can save time, improve migration efficiency, and automatically detect the migration effect after the database migration to ensure migration accuracy. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart illustrating the migration operation performed by the Redis database migration system provided in this embodiment of the invention. Detailed Implementation

[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] This invention provides a Redis database migration system for automatically migrating Redis databases. The system can be implemented on an existing operating system such as Linux, using a software platform that can be either a browser / server (B / S) or client / server (C / S) architecture. Specifically, the Redis database migration system provided by this invention may include a processor and a memory connected by communication. The memory stores a configuration file. The processor is connected to one or more display devices to execute a computer program that displays a configuration page on the display device based on the configuration file. The configuration page allows the user to configure migration information, which includes at least the database ID, environment information, connection port, data to be migrated from the source database, the required migration threads, migration tools, detection tools, number of detections, and detection accuracy.

[0033] In this embodiment of the invention, the database environment information may include at least the architecture, version, data size, data capacity, business scenario, and operating system version. The migration information acquisition module is used to acquire migration information, which includes at least the data to be migrated from the source database, the threads required for migration, migration tools, detection tools, number of detections, and detection accuracy.

[0034] In this embodiment of the invention, the data to be migrated may include part of the data in the source database, such as a certain number of data entries in the database, and all of the data, i.e., the entire data in the source database.

[0035] In this embodiment of the invention, the number of detections can be set based on actual needs. In one illustrative embodiment, the maximum number of detections is 3. The detection accuracy may include the ID of the detection data, the data value of the detection data, or both the ID and data value of the detection data. The data ID is the data number. Different detection accuracies result in different computational loads, with the computational load being the highest when detecting both the ID and data value of the data simultaneously.

[0036] In embodiments of the present invention, the migration tool may include at least Redis-dts. In one illustrative embodiment, it may include Redis-dts and data sequencing filtering tools such as fastp, AfterQC, SOAPnuke, seqtk, Trimmomatic, QcReads, cutadapt, fastx_toolkit, etc.

[0037] In addition, a task start button is also provided on the configuration page.

[0038] In practical applications, users can log in to the configuration page with their corresponding account and password, enter the basic and migration information required for the migration task, and then start the migration task by clicking the task start button on the configuration page.

[0039] Specifically, in response to detecting that a user has clicked the task start button, the processor executes a computer program to perform, for example... Figure 1 The steps shown are as follows:

[0040] S100, based on the obtained source database and target database IDs and connection ports, selects the corresponding docking ports to establish communication connections with the corresponding source database and target database respectively.

[0041] In this embodiment of the invention, the interface port includes at least the middleware port and the redis-server port, thereby enabling adaptation to various types of databases.

[0042] S200: Based on the obtained number of threads required for migration, allocate corresponding threads to the current migration task, allocate the amount of data to be migrated to each thread based on the obtained data to be migrated, and select the corresponding migration method for the current migration task based on the obtained environment information.

[0043] In this embodiment of the invention, allocating the amount of data to be migrated to each thread based on the data to be migrated can be an existing method. For example, if the amount of data to be migrated is divisible by the number of threads, it can be evenly allocated to each thread; if it is not divisible, it can be roughly evenly allocated.

[0044] 0. In this embodiment of the invention, the migration methods provided by the system may include at least a smooth migration method, a heterogeneous migration method, and a persistent migration method. The smooth migration method uses dual-write data to achieve smooth migration between databases, enabling seamless data migration. The heterogeneous migration method supports online data migration between heterogeneous Redis clusters, meaning that the source cluster can still accept business read and write requests normally during the data migration process, thus ensuring uninterrupted service.

[0045] It supports data migration between various heterogeneous clusters and scaling up / down of homogeneous clusters, including migration from a single Redis instance to a 5-twemproxy / cluster, migration from twemproxy to twemproxy / cluster, and migration from cluster to twemproxy / cluster. Persistence migration methods ensure the persistence of the Redis database by synchronizing data stored in memory to disk. These methods primarily include RDB and AOF. RDB persistence generates point-in-time snapshots of the dataset at specified time intervals; AOF persistence records all write operation commands executed by the server and restores the dataset by re-executing these commands when the server starts.

[0046] 0S300 retrieves the corresponding data from the source database based on the acquired data that needs to be migrated, performs backup, obtains the backup data, and allocates it to the corresponding thread.

[0047] S400: For each thread, the migration tool corresponding to the current migration task is used to migrate the corresponding data to the target database based on the selected migration method. During the migration process, the changed data in the source database is acquired in real time and stored.

[0048] 5. In this embodiment of the invention, the changed data in the source database may include modified data that modifies the original data and new data newly added to the database. The changed data may be stored in a cache area.

[0049] S500: After detecting that the backup data migration is complete, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, and migrate it to the target database; execute S600.

[0050] In this embodiment of the invention, if the data to be migrated is a portion of the data in the source database, then the changed data corresponding to the data to be migrated is the data that has changed in the portion of the data. If the data to be migrated is all the data in the source database, then the changed data corresponding to the data to be migrated includes the data that changed in the previous migration and the data newly added to the current source database.

[0051] S600: If no data needs to be migrated in the source database, execute S700; otherwise, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, migrate it to the target database, and execute S600.

[0052] In one illustrative embodiment of the present invention, if a user-inputted instruction to end the migration task is received, it indicates that there is no data in the source database that needs to be migrated further. In another illustrative embodiment, if no changes are detected in the data in the source database within a set time period, it indicates that there is no data in the source database that needs to be migrated further.

[0053] OS700, based on the acquired number of detections and detection accuracy, utilizes the detection tool corresponding to the current migration task to analyze the source database.

[0054] The system checks the consistency between the data to be migrated and the data to be migrated into the target database. If they are consistent, the control program exits; otherwise, it executes S800.

[0055] In this embodiment of the invention, existing data consistency detection methods, such as cosine similarity, can be used to detect data consistency. The specific detection process can be based on existing technologies.

[0056] The 5S800 retrieves inconsistent data from the source database, assigns it to the corresponding thread, and migrates it to the target database.

[0057] And exit the control program after migration.

[0058] Furthermore, in this embodiment of the invention, if data loss in the cache is detected, indicating that the data in the source database and the target database are inconsistent, the control program exits and sends a message to the user to restart the migration task, thereby restarting the migration task operation.

[0059] 0. Furthermore, the processor is also configured to execute a computer program to perform the following steps:

[0060] If problems arise during the migration process, obtain the corresponding solutions and handle them.

[0061] Specifically, in this embodiment of the invention, the provided system also pre-stores a problem solution information table. The i-th row of the problem solution information table includes problem i and the solution to problem i, where i ranges from 1 to n, and n is the number of problems.

[0062] All issues listed in the problem information table represent all possible problems that may arise during the migration process, obtained from historical data. For example, many errors may occur in migration process 5, such as: the Linux firewall might block access to ports outside the configuration file; the solution is to add the port to ` / etc / sysconfig / iptables` in the target Linux system; or a key in the target Redis instance might be identical to the key to be migrated; the solution is to add the `replace` parameter to the command, etc. By setting up a problem solution information table, when problems occur during the migration process, the corresponding solutions can be obtained by querying the table, thus enabling automatic and rapid resolution.

[0063] The Redis database migration system provided in this embodiment of the invention can automatically detect the type and configuration of the database during the Redis database migration process, automatically design a migration plan according to the migration requirements, and automatically load the corresponding software tools to perform the data migration operation. This can save time, improve migration efficiency, and automatically detect the migration effect after the database migration, ensuring migration accuracy.

[0064] Based on the same inventive concept, another embodiment of the present invention provides a Redis database migration method, the method comprising:

[0065] S10, Obtain migration information, which includes at least the database ID, environment information, connection port, data to be migrated from the source database, the threads required for migration, migration tools, detection tools, number of detections, and detection accuracy;

[0066] S20, perform the corresponding migration operation based on the acquired migration information;

[0067] S20 specifically includes:

[0068] S201, Based on the obtained source database and target database IDs and connection ports, select the corresponding docking ports to establish communication connections with the corresponding source database and target database respectively;

[0069] S202, based on the obtained number of threads required for migration, allocate the corresponding threads to the current migration task, based on the obtained data to be migrated, allocate the amount of data to be migrated to each thread, and select the corresponding migration method for the current migration task based on the obtained environment information;

[0070] S203: Based on the acquired data that needs to be migrated, retrieve the corresponding data from the corresponding source database for backup, obtain the backup data, and allocate it to the corresponding thread;

[0071] S204, For each thread, the migration tool corresponding to the current migration task is used to migrate the corresponding data to the target database based on the selected migration method. During the migration process, the changed data in the source database is acquired in real time and stored.

[0072] S205, after detecting that the backup data migration is complete, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, and migrate it to the target database; execute S206;

[0073] S206, if no data needs to be migrated in the source database, execute S207; otherwise, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, migrate it to the target database, and execute S206.

[0074] S207: Based on the obtained number of detections and detection accuracy, use the detection tool corresponding to the current migration task to detect the consistency between the data to be migrated in the source database and the data to be migrated into the target database. If they are consistent, exit the control program; otherwise, execute S208.

[0075] S208: Obtain inconsistent data from the source database, assign it to the corresponding thread, migrate it to the target database, and exit the control program after migration.

[0076] The implementation of the above steps can be found in the foregoing embodiments. To avoid redundancy, their specific descriptions are omitted.

[0077] Embodiments of the present invention also provide a non-transitory computer-readable storage medium that can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a method in the method embodiments, wherein the at least one instruction or the at least one program is loaded and executed by the processor to implement the method provided in the above embodiments.

[0078] Embodiments of the present invention also provide an electronic device, including a processor and the aforementioned non-transitory computer-readable storage medium.

[0079] Embodiments of the present invention also provide a computer program product including program code, which, when the program product is run on an electronic device, causes the electronic device to perform the steps of the methods described above in various exemplary embodiments of the present invention.

[0080] While specific embodiments of the invention have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of the invention. Those skilled in the art should also understand that various modifications can be made to the embodiments without departing from the scope and spirit of the invention. The scope of this invention is defined by the appended claims.

Claims

1. A Redis database migration system, characterized in that, The system includes: a processor and a memory connected by communication; the memory stores a configuration file and a pre-stored problem solution information table; the i-th row of the problem solution information table includes problem i and its solution, where i ranges from 1 to n, and n is the number of problems; the processor is connected to one or more display devices to execute a computer program that displays a configuration page on the display device based on the configuration file; the configuration page allows the user to configure migration information, which includes at least the database ID, environment information, connection port, data to be migrated from the source database, the threads required for migration, migration tools, detection tools, number of detections, and detection accuracy; the migration tools include Redis-dts and data sequencing filtering tools. The processor is also configured to execute computer programs to perform the following steps: S100: Based on the obtained source database and target database IDs and connection ports, select the corresponding docking ports to establish communication connections with the corresponding source database and target database respectively; S200: Based on the obtained number of threads required for migration, allocate corresponding threads to the current migration task and allocate the amount of data to be migrated to each thread based on the obtained data to be migrated; and select the corresponding migration method for the current migration task based on the obtained environment information; the migration method includes at least smooth migration method, heterogeneous migration method and persistent migration method. S300 retrieves the corresponding data from the source database based on the acquired data that needs to be migrated, performs backup, obtains the backup data, and allocates it to the corresponding thread; S400, for each thread, the migration tool corresponding to the current migration task is used to migrate the corresponding data to the target database based on the selected migration method. During the migration process, the changed data in the source database is acquired in real time and stored. The changed data is stored in the cache area. S500: After detecting that the backup data migration is complete, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, and migrate it to the target database; execute S600; S600: If no data needs to be migrated in the source database, proceed to S700; otherwise, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, migrate it to the target database, and proceed to S600. S700: Based on the number of detections and the detection accuracy, use the detection tool corresponding to the current migration task to check the consistency between the data to be migrated in the source database and the data to be migrated into the target database. If they are consistent, exit the control program; otherwise, execute S800. S800: Obtain inconsistent data from the source database, assign it to the corresponding thread, migrate it to the target database, and exit the control program after migration; The processor is also configured to execute computer programs to perform the following steps: If data loss in the cache is detected, the control program exits and sends a message to the user to restart the migration task. If problems occur during the migration process, query the problem solution information table to obtain the corresponding solution and handle it.

2. The system according to claim 1, characterized in that, The data that needs to be migrated includes some and all of the data in the source database.

3. The system according to claim 2, characterized in that, In S400, if the data to be migrated is a portion of the data in the source database, then the changed data corresponding to the data to be migrated is the data that has changed in the portion of the data. If the data to be migrated is all the data in the source database, then the changed data corresponding to the data to be migrated includes the data that changed in the previous migration and the data newly added to the current source database.

4. The system according to claim 1, characterized in that, The detection accuracy includes the ID of the detection data, the data value of the detection data, and the ID and data value of the detection data.

5. The system according to claim 1, characterized in that, The database environment information includes at least the architecture, version, data size, data capacity, business scenario, and operating system version.

6. The system according to claim 1, characterized in that, The interface ports include at least the middleware port and the redis-server port.

7. A Redis database migration method, characterized in that, The method includes: S10, Obtain migration information, which includes at least the database ID, environment information, connection port, data to be migrated from the source database, the threads required for migration, migration tools, detection tools, number of detections, and detection accuracy; the migration tools include Redis-dts and data sequencing filtering tools. S20, Perform the corresponding migration operation based on the acquired migration information; S20 specifically includes: S201, Based on the obtained source database and target database IDs and connection ports, select the corresponding docking ports to establish communication connections with the corresponding source database and target database respectively; S202, based on the obtained number of threads required for migration, allocate corresponding threads to the current migration task and allocate the amount of data to be migrated to each thread based on the obtained data to be migrated, and select the corresponding migration method for the current migration task based on the obtained environment information; the migration method includes at least smooth migration method, heterogeneous migration method and persistent migration method; S203, based on the obtained data to be migrated, obtain the corresponding data from the corresponding source database, back it up, obtain the backup data, and allocate it to the corresponding thread; S204, For each thread, the migration tool corresponding to the current migration task is used to migrate the corresponding data to the target database based on the selected migration method. During the migration process, the changed data in the source database is acquired in real time and stored. The changed data is stored in the cache. S205, After the backup data migration is detected to be complete, the changed data corresponding to the data to be migrated is retrieved from the stored data and assigned to the corresponding thread and migrated to the target database. Execute S206. S206, if no data needs to be migrated in the source database, execute S207; otherwise, retrieve the changed data corresponding to the data to be migrated from the stored data, allocate it to the corresponding thread, migrate it to the target database, and execute S206. S207: Based on the obtained number of detections and detection accuracy, use the detection tool corresponding to the current migration task to detect the consistency between the data to be migrated in the source database and the data to be migrated into the target database. If they are consistent, exit the control program; otherwise, execute S208. S208: Obtain inconsistent data from the source database, assign it to the corresponding thread, migrate it to the target database, and exit the control program after migration; The i-th row of the problem solution information table includes problem i and the solution to problem i, where i ranges from 1 to n and n is the number of problems; The method further includes: If data loss in the cache is detected, the control program exits and sends a message to the user to restart the migration task. If problems occur during the migration process, query the problem solution information table to obtain the corresponding solution and handle it.

8. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the processor is used to execute the at least one instruction or the at least one program to implement the method of claim 7.

Images