Data migration method and device, electronic equipment and storage medium

Abstract

The application discloses a data migration method and device, electronic equipment and a storage medium. The method comprises the following steps: in the case that a user request to be sent to a first application is monitored, the user request is sent to the first application and the user request is copied, wherein the user request carries identification information, and the identification information is used for indicating reading of the user request or migration of the user request; according to the identification information, a second application reads the copied user request or sends the copied user request to a third application and writes the copied user request into a target database corresponding to the third application, and the target database comprises full data of a source database corresponding to the first application. The embodiment realizes continuity of business in the process of data migration.

Description

Technical Field

[0001] This application belongs to the field of data migration technology, specifically relating to a data migration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Data migration is a challenging aspect of application system restructuring and technological transformation. Currently, in database migration scenarios, data from the source database is typically migrated to the target database in one go, but this requires a business downtime window, the size of which depends on the time required for a full data migration.

[0003] In the above solution, when the data volume is large, the entire migration process may take several days, which results in long business downtime and cannot guarantee business continuity. Summary of the Invention

[0004] This application provides a data migration method, apparatus, electronic device, and storage medium to solve the problem that the data migration process cannot guarantee business continuity in related technologies.

[0005] In a first aspect, embodiments of this application provide a data migration method, including:

[0006] Upon detecting a user request to be sent to a first application, the user request is sent to the first application and the user request is copied, wherein the user request carries identification information, which is used to indicate whether the user request is read or migrated.

[0007] Based on the identification information, the second application is controlled to read the copied user requests or send the copied user requests to the third application and write them into the target database corresponding to the third application. The target database includes all the data of the source database corresponding to the first application.

[0008] Secondly, embodiments of this application further provide a data migration apparatus, including:

[0009] The sending module is configured to send the user request to the first application and copy the user request when a user request to be sent to the first application is detected, wherein the user request carries identification information, and the identification information is used to indicate whether the user request is read or the user request is migrated.

[0010] The processing module is configured to, based on the identification information, control the second application to read the copied user request or send the copied user request to the third application and write it into the target database corresponding to the third application, wherein the target database includes all data of the source database corresponding to the first application.

[0011] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0012] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0013] The solution provided in this application, upon detecting a user request to be sent to a first application, sends the user request to the first application and copies the user request. The user request carries identification information, which indicates whether the user request should be read or migrated. Based on the identification information, a second application is controlled to read the copied user request, or send the copied user request to a third application and write it to a target database corresponding to the third application. The target database includes all data from the source database corresponding to the first application. This achieves the simultaneous transmission of user requests to the first application for processing, copying user requests, and reading or writing the copied user requests to the target database as needed. This ensures the continuity of business processing while migrating incremental data to the target database, solving the problem of business continuity not being guaranteed during data migration. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the data migration method in an embodiment of this application;

[0015] Figure 2 This is a schematic diagram of the data migration method in the embodiments of this application;

[0016] Figure 3 This is a schematic diagram of the data migration device in the embodiments of this application;

[0017] Figure 4 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0020] The data migration method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0021] Figure 1 An embodiment of the present invention provides a data migration method that can be performed by an electronic device, which may include a server and / or a terminal device. In other words, the method can be performed by software or hardware installed on the electronic device, and the method includes the following steps:

[0022] Step 101: If a user request to be sent to the first application is detected, the user request is sent to the first application and the user request is copied.

[0023] Specifically, a user request can be a Hypertext Transfer Protocol (HTTP) request.

[0024] Upon detecting a user request to be sent to the first application, the user request can be sent to the first application, thereby enabling the first application to process the user request. The first application corresponds to a source database, which stores all data up to the current time.

[0025] Furthermore, when sending a user request to the first application, it is also necessary to copy the user request, which carries identification information indicating whether the user request should be read or migrated. This allows the copied user request to be read or migrated based on the identification information carried by the user request.

[0026] Step 102: Based on the identification information, control the second application to read the copied user request, or send the copied user request to the third application and write it into the target database corresponding to the third application.

[0027] Specifically, if the identification information indicates that a user request should be read, the second application can be controlled to read the copied user request; if the identification information indicates that a user request should be migrated, the copied user request can be sent to the third application and written to the target database corresponding to the third application, thereby achieving a smooth migration of new user requests to the target database without the business being aware of it. Furthermore, the different information indicated by the identification information enables the data read or write processing, meeting different business needs.

[0028] In addition, the second application can be a test application, which of course has a corresponding test database. In this case, the second application can be used to perform stress testing or functional verification of the system.

[0029] The third application and the first application can have the same business attributes, which allows the request results returned by the third application to be the same as those returned by the first application, thereby enabling the reconstruction of the application system.

[0030] It should be noted that the target database includes the full data of the source database corresponding to the first application, which ensures the data integrity of the target database after new user requests are migrated to the target database.

[0031] In this embodiment, upon detecting a user request to be sent to the first application, the user request is sent to the first application and copied. Based on the identification information carried in the user request, the second application is controlled to read the copied user request, or the copied user request is sent to a third application and written to the target database corresponding to the third application. The target database includes all data from the source database corresponding to the first application. This achieves the simultaneous and uninterrupted sending of user requests to the first application for processing, copying of user requests, and reading of the copied user requests as needed.

[0032] Alternatively, the data can be written to the target database, thereby migrating the new data to the target database while ensuring the continuity of business processing and solving the problem that the data migration process cannot guarantee business continuity.

[0033] Furthermore, this embodiment can implement the user request replication process using the application layer's OpenResry+Lua+Kafka approach. Specifically, in one embodiment, sending the user request to the first application and replicating the user request includes the following steps:

[0034] The user request is sent to the first application via OpenResty, and then sent to a first topic or a second topic in Kafka according to the identification information. The first topic is used to store user requests to be read, and the second topic is used to store user requests to be migrated.

[0035] Specifically, OpenResty is a high-performance global wide area network (WWAN) platform based on Nginx and Lua. It integrates a large number of excellent Lua libraries, third-party modules, and most dependencies, making it easy to build dynamic web applications, web services, and dynamic gateways that can handle ultra-high concurrency and have high scalability.

[0036] In the Nginx system, this embodiment can configure OpenResry's `ngx.location.capture_multi` to achieve concurrent replication of the same user request. Specifically, this embodiment can be configured in the OpenResry configuration file...

[0037] The relevant parameters are set in the application, such as connection timeout, send timeout, read timeout, timeout to maintain 0 high availability, connection pool size, cache size, etc., and the header and body of the user request are obtained. The capture_multi API is used to copy the request body and forward the user request to the first application.

[0038] Furthermore, this embodiment can also create Kafka producers for OpenResry and define producer consumption.

[0039] The system generates a message format and a Kafka cluster broker list, and creates a first Topic for storing user requests to be read and a second Topic for storing user requests to be migrated. This allows the embodiment to send user requests to the first application via OpenResty, and to send user requests indicating reads with identification information to the first Topic in Kafka via OpenResty, and user requests indicating migration with identification information to the second Topic, thereby realizing the data replication process through OpenResty + Lua + Kafka.

[0040] Furthermore, in one embodiment, based on the identification information, controlling the second application to read the copied user request or send the copied user request to the third application and write it into the target database corresponding to the third application includes:

[0041] The second application is controlled to read the user request from the first topic, or to send the user request from the second topic to the third application and write it into the target database corresponding to the third application.

[0042] Specifically, since the first Topic is used to store user requests to be read and the second Topic is used to store user requests to be migrated, the second application can be controlled to read user requests from the first Topic or send user requests from the second Topic to the third application and write them to the target database corresponding to the third application. This enables the retrieval of user requests of different instruction types from different topics, avoiding the writing of user requests to be read to the target database or the reading of user requests to be migrated to the second application.

[0043] In addition, in order to ensure the processing order of requests with the same business primary key, in one embodiment, both the first Topic and the second Topic are pre-partitioned according to the business primary key information, and the user request also carries the user's business primary key information;

[0044] At this time, sending the user request to the first topic or the second topic in Kafka according to the identification information includes: sending the user request to the partition corresponding to the business primary key information in the first topic or the second topic according to the business primary key information.

[0045] Specifically, business primary key information may include user identifier, user business order number, etc., and of course, the business primary key information corresponding to each user is unique.

[0046] By sending user requests to the partition corresponding to the business key information in the first or second topic based on the business key information, each partition stores only one business request corresponding to the business key. This ensures the processing order of requests with the same business key by leveraging the first-in-first-out characteristic of messages within the partition.

[0047] The following is through Figure 2 This embodiment will be described in detail.

[0048] like Figure 2 As shown, assuming the application system is an nginx system, when the nginx system receives a user request, it can send the user request to the first application through OpenResty. Of course, at this time, the first application can query the request result of the user request from the source database.

[0049] This embodiment can create an OpenResty Kafka producer, specify the producer's message format, the Kafka cluster broker list, and the Kafka consumer. It also creates a first topic for storing user requests to be read and a second topic for storing user requests to be migrated. Each user request carries identification information indicating whether it should be read or migrated. OpenResty can then copy the user request to either the first or second topic in Kafka. Based on the identification information, the second application can be controlled to read the user request from the first topic, or, through the Kafka message consumer, call the third application's application program interface (API) to send the user request from the second topic to the third application and write it to the target database corresponding to the third application. Specifically, the second application can be a test application with a corresponding test database, thereby enabling stress or functional testing of the reconstructed application system.

[0050] It should be noted that the target database includes all the data in the source database, thus ensuring the integrity of the target database.

[0051] In this way, this embodiment realizes the simultaneous sending of user requests to the first application for processing without interruption, and the copying of user requests to the target database as needed. This enables the migration of incremental data to the target database while ensuring the continuity of business processing. It can automatically adapt to changes in data models and optimization of application logic during application system reconstruction and data migration, thereby solving the problems of data link redundancy and duplicate data processing logic during data migration.

[0052] Figure 3 This diagram illustrates the structure of a data migration apparatus according to an embodiment of the present invention. Figure 3 As shown, the data migration device includes:

[0053] The sending module 301 is used to send the user request to the first application and copy the user request when a user request to be sent to the first application is detected, wherein the user request carries identification information, and the identification information is used to indicate whether the user request is read or the user request is migrated.

[0054] The processing module 302 is used to control the second application to read the copied user request according to the identification information, or to send the copied user request to the third application and write it into the target database corresponding to the third application, wherein the target database includes the full data of the source database corresponding to the first application.

[0055] In one embodiment, the sending module 301 is configured to send the user request to the first application via OpenResty, and send the user request to Kafka according to the identification information.

[0056] The first topic or the second topic, wherein the first topic is used to store user requests to be read, and the second topic is used to store user requests to be migrated.

[0057] In one embodiment, both the first Topic and the second Topic are pre-partitioned according to business primary key information, and the user request also carries the user's business primary key information;

[0058] The sending module is used to send the user request to the partition corresponding to the business key information in the first topic or the second topic, according to the business key information.

[0059] 5. In one embodiment, the processing module is configured to control the second application to read the user request from the first topic, or to send the user request in the second topic to the third application and write it into the target database corresponding to the third application.

[0060] The data migration apparatus provided in this application embodiment can achieve... Figure 1-2 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0061] It should be noted that the embodiments of the data migration device in this specification and the embodiments of the data migration method in this specification are based on the same inventive concept. Therefore, for specific implementation of the data migration device embodiments, please refer to the corresponding implementation of the data migration method embodiments mentioned above. Repeated descriptions will not be repeated.

[0062] The data migration device in this application embodiment can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, mobile electronic devices can be mobile phones, tablets, laptops, PDAs, in-vehicle electronic devices, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc., while non-mobile electronic devices can be servers, network-attached storage (NAS), personal computers (PCs), televisions (TVs), ATMs, or self-service machines, etc. This application embodiment does not impose specific limitations.

[0063] The data migration device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0064] Based on the same technical concept, embodiments of this application also provide an electronic device for performing the above-described data migration method. Figure 4 This is a schematic diagram of the structure of an electronic device to implement various embodiments of this application. The electronic device can vary significantly due to differences in configuration or performance, and may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call a computer program stored in the memory 430 and executable on the processor 410 to perform the following steps:

[0065] Upon detecting a user request to be sent to a first application, the user request is sent to the first application and the user request is copied, wherein the user request carries identification information, which is used to indicate whether the user request is read or migrated.

[0066] Based on the identification information, the second application is controlled to read the copied user requests, or to send the copied user requests to the third application and write them into the target database corresponding to the third application. The target database includes all the data of the source database corresponding to the first application.

[0067] In one embodiment, sending the user request to the first application and replicating the user request includes: sending the user request to the first application via OpenResty, and sending the user request to a first topic or a second topic in Kafka according to the identification information, wherein the first topic is used to store user requests to be read, and the second topic is used to store user requests to be migrated.

[0068] In one embodiment, both the first Topic and the second Topic are pre-partitioned according to business primary key information, and the user request also carries the user's business primary key information;

[0069] Sending the user request to the first or second topic in Kafka according to the identification information includes: sending the user request to the partition corresponding to the business primary key information in the first or second topic according to the business primary key information.

[0070] In one embodiment, controlling the second application to read the copied user request or send the copied user request to the third application and write it to the target database corresponding to the third application, based on the identification information, includes: controlling the second application to read the user request from the first topic or send the user request from the second topic to the third application and write it to the target database corresponding to the third application, based on the identification information.

[0071] The specific execution steps can be found in the various steps of the above data migration method embodiments, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0072] It should be noted that the electronic devices in the embodiments of this application include: servers, terminals, or other devices besides terminals.

[0073] The above electronic device structure does not constitute a limitation on the electronic device. An electronic device may include more or fewer components than illustrated, or combine certain components, or arrange them differently. For example, an input unit may include a Graphics Processing Unit (GPU) and a microphone, and a display unit may use a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar display panels. User input units include at least one of a touch panel and other input devices. A touch panel is also called a touchscreen. Other input devices may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be elaborated further here.

[0074] Memory can be used to store software programs and various data. Memory can primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area can store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, memory can include volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (Synchlink DRAM, SLDRAM), and direct memory bus RAM (DRRAM).

[0075] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly handles operations related to the operating system, user interface, and applications, while the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor.

[0076] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described data migration method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0077] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0078] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above method embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.

[0079] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0080] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0082] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A data migration method, characterized by, The application comprises the following steps: In the case of monitoring a user request to be sent to a first application, the user request is sent to the first application and copied, wherein the user request carries identification information, and the identification information is used to indicate reading of the user request or migration of the user request; According to the identification information, a second application reads the copied user request, or the copied user request is sent to a third application and written into a target database corresponding to the third application, and the target database comprises full data of a source database corresponding to the first application; The step of sending the user request to the first application and copying the user request comprises the following steps: The user request is sent to the first application through OpenResty, and the user request is sent to a first topic Topic or a second Topic in Kafka according to the identification information, wherein the first Topic is used to store the user request to be read, and the second Topic is used to store the user request to be migrated.

2. The data migration method of claim 1, wherein, The first Topic and the second Topic are pre-partitioned according to business primary key information, and the user request further carries business primary key information of a user; The step of sending the user request to a first topic Topic or a second Topic in Kafka according to the identification information comprises the following step: The user request is sent to a partition corresponding to the business primary key information in the first Topic or the second Topic according to the business primary key information.

3. The data migration method of claim 1, wherein, According to the identification information, the second application reads the copied user request, or the copied user request is sent to a third application and written into a target database corresponding to the third application, and the target database comprises full data of a source database corresponding to the first application. According to the identification information, the second application reads the user request from the first Topic, or the user request in the second Topic is sent to a third application and written into a target database corresponding to the third application.

4. A data migration apparatus, characterized by comprising: The application comprises the following steps: A sending module is configured to send a user request to a first application and copy the user request in the case of monitoring the user request to be sent to the first application, wherein the user request carries identification information, and the identification information is used to indicate reading of the user request or migration of the user request; A processing module is configured to control a second application to read the copied user request according to the identification information, or send the copied user request to a third application and write the copied user request into a target database corresponding to the third application, and the target database comprises full data of a source database corresponding to the first application; The sending module is configured to send the user request to the first application through OpenResty, and send the user request to a first topic Topic or a second Topic in Kafka according to the identification information, wherein the first Topic is used to store the user request to be read, and the second Topic is used to store the user request to be migrated.

5. The data migration apparatus of claim 4, wherein, The first Topic and the second Topic are both pre-partitioned according to business primary key information, and the user request also carries the business primary key information of the user; The sending module is configured to send the user request to a partition corresponding to the business primary key information in the first Topic or the second Topic according to the business primary key information.

6. The data migration apparatus of claim 4, wherein, The processing module is configured to control the second application to read the user request from the first Topic, or send the user request in the second Topic to a third application and write the user request into a target database corresponding to the third application.

7. An electronic device, comprising: A processor, a memory, and a program or instruction stored on the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the data migration method according to any one of claims 1-3.

8. A readable storage medium, characterized by, A readable storage medium storing a program or instruction, wherein the program or instruction, when executed by a processor, implements the steps of the data migration method according to any one of claims 1-3.

Images