Data processing method and device, equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, equipment, a storage medium and a program product, which are applied to various data migration scenes such as cloud technology, artificial intelligence, intelligent transportation, information processing, games, maps and vehicle-mounted data migration scenes. The method comprises the following steps: performing data backup on a source database to obtain a database mirror image; performing database recovery in a specified database based on the database mirror image to obtain an initial database; first change data in the source database is updated to the initial database to obtain a to-be-switched database, and the first change data is data, changed during database recovery, of the source database; switching a database for business processing from a source database to a database to be switched; second change data in the source database is updated to the to-be-switched database, a target database is obtained, and the second change data is data of the source database changed during the updating period of the first change data; and performing business processing based on the target database. Through the method, the data migration efficiency can be improved.

Description

Technical Field

[0001] The present application relates to database technology in the field of computer applications, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] In the process of data processing, cross-database data migration is often involved. Generally speaking, in order to achieve cross-database data migration, data migration is usually carried out by rebuilding data indexes; that is, rebuilding the data index of the source database in the new database to migrate the data in the data index of the source database to the data index of the new database. However, in the above cross-database data migration process, the speed of data migration by rebuilding data indexes is lower than the specified threshold, which affects the efficiency of data migration. Summary of the invention

[0003] The embodiments of the present application provide a data processing method, apparatus, device, storage medium and program product, which can improve data migration efficiency.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present invention provides a data processing method, which includes:

[0006] Back up the source database to obtain a database image;

[0007] Performing database recovery in a designated database based on the database mirror to obtain an initial database;

[0008] Updating the first changed data in the source database to the initial database to obtain the database to be switched, wherein the first changed data is the data changed during database recovery;

[0009] Switching the database for business processing from the source database to the database to be switched;

[0010] In the process of performing business processing based on the database to be switched, the second changed data in the source database is updated to the database to be switched to obtain a target database, wherein the second changed data is data changed during the update of the first changed data;

[0011] Business processing is performed based on the target database.

[0012] The present application provides a data processing device, the data processing device comprising:

[0013] The image acquisition module is used to back up the source database and obtain the database image;

[0014] A data recovery module, used for performing database recovery in a designated database based on the database mirror to obtain an initial database;

[0015] An incremental update module, configured to update first changed data in the source database to the initial database to obtain a database to be switched, wherein the first changed data is data changed during database recovery;

[0016] A business processing module, used to switch a database for business processing from the source database to the database to be switched;

[0017] The incremental update module is further used to update the second changed data in the source database to the database to be switched to obtain a target database during business processing based on the database to be switched, wherein the second changed data is data changed during the update of the first changed data;

[0018] The business processing module is also used to perform business processing based on the target database.

[0019] In an embodiment of the present application, the incremental update module is also used to perform a first index migration on the first data to be updated in the source database in the initial database to obtain a first intermediate database, wherein the first change data includes the first data to be updated updated during the recovery of the database; and delete the first data to be deleted in the source database from the first intermediate database to obtain the database to be switched, wherein the first change data also includes the first data to be deleted deleted during the recovery of the database.

[0020] In an embodiment of the present application, the incremental update module is also used to determine, in the source database, a first source data index of the first data to be updated; in the initial database, determine a first target data index corresponding to the first source data index; and execute a first index migration of data from the first source data index to the first target data index to obtain the first intermediate database.

[0021] In an embodiment of the present application, the incremental update module is also used to obtain the target access log of the source database during the database recovery period; parse the target deletion operation in the target access log, and the target deletion operation is used to delete the first data to be deleted; execute the target deletion operation in the first intermediate database to obtain the database to be switched.

[0022] In an embodiment of the present application, the data processing device further includes a field adding module, which is used to add a latest update time field in the source database, and the latest update time field is used to record the latest update time of the data.

[0023] In the embodiment of the present application, the data processing device further includes a data determination module, which is used to determine, in the source database, data whose latest update time is changed during the database recovery period as the first changed data.

[0024] In the embodiment of the present application, the data determination module is further used to determine, in the source database, data whose latest update time is changed during the update period of the first changed data as the second changed data.

[0025] In an embodiment of the present application, the incremental update module is also used to perform a second index migration on the second data to be updated in the source database in the database to be switched, to obtain a second intermediate database, wherein the second change data includes the second data to be updated updated during the update of the first change data; and delete the second data to be deleted in the source database from the second intermediate database to obtain the target database, wherein the second change data includes the second data to be deleted deleted during the update of the first change data.

[0026] In an embodiment of the present application, the incremental update module is also used to determine, in the database to be switched, the target data version corresponding to the second data to be updated in the source database; in the source database, determine the source data version corresponding to the second data to be updated; from the second data to be updated, filter the target data to be updated whose target data version is smaller than the source data version; in the database to be switched, perform a second index migration on the target data to be updated to obtain the second intermediate database.

[0027] In an embodiment of the present application, the image acquisition module is also used to connect to the source database based on the source database address, and the source database address is used to access the source database; create a backup warehouse for the source database; in the backup warehouse, create a snapshot based on the index of the data to be backed up in the source database to obtain the database image.

[0028] In an embodiment of the present application, the data recovery module is also used to connect to the specified database based on a specified database address, and the specified database address is used to access the specified database; in the specified database, the backup warehouse of the source database is associated; and the database image in the backup warehouse is snapshot restored to obtain the initial database.

[0029] In the embodiment of the present application, the business processing module is further used to determine the initial database as a database for verifying business processing.

[0030] In the embodiment of the present application, the business processing module is further used to switch the database of the business processing from the source database to the database to be switched when it is determined that the verification business processing passes during the process of verifying the business processing based on the initial database.

[0031] In the embodiment of the present application, the business processing module is further used to delete the source database when it is determined that the business processing is passed during the business processing based on the target database.

[0032] An embodiment of the present application provides an electronic device for data processing, the electronic device comprising:

[0033] A memory for storing computer executable instructions or computer programs;

[0034] The processor is used to implement the data processing method provided in the embodiment of the present application when executing the computer executable instructions or computer programs stored in the memory.

[0035] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program are used to implement the data processing method provided in the embodiment of the present application when executed by a processor.

[0036] An embodiment of the present application provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the data processing method provided by the embodiment of the present application is implemented.

[0037] The embodiments of the present application have at least the following beneficial effects: when performing data migration across databases on a source database, the source database is first restored in a designated database through a data backup method using database mirroring; then incremental updates are performed before and after the database switch to complete the switch of the business processing database from the source database to the target database; since the database recovery speed of the data backup is greater than the incremental update speed, the database is first restored through a data backup method to complete the migration of the full amount of data in the source data; then incremental updates are performed to complete the data update of the source data after the data backup; and data migration efficiency can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is an exemplary schematic diagram of migration time consumption;

[0039] Figure 2 It is a schematic diagram that leads to dirty data;

[0040] Figure 3 It is a schematic diagram of an update coverage;

[0041] Figure 4 It is a schematic diagram of the architecture of the data processing system provided in the embodiment of the present application;

[0042] Figure 5 This is a method provided by the embodiment of the present application. Figure 4 A schematic diagram of the structure of the terminal in FIG.

[0043] Figure 6 This is a flow chart of the data processing method provided in the embodiment of the present application. Figure 1 ;

[0044] Figure 7 This is a flow chart of the data processing method provided in the embodiment of the present application. Figure 2 ;

[0045] Figure 8 is an exemplary application schematic diagram provided in an embodiment of the present application;

[0046] Fig. 9 is another exemplary application schematic diagram provided in an embodiment of the present application;

[0047] Fig.10 is an exemplary load schematic diagram provided in an embodiment of the present application;

[0048] Fig.11 It is a schematic diagram of a data migration method provided in an embodiment of the present application;

[0049] Fig.12 It is an exemplary overall architecture diagram provided by an embodiment of the present application;

[0050] Fig.13 It is another exemplary load schematic diagram provided in an embodiment of the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0052] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0053] In the following description, the terms "first\second" are used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0054] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0055] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0056] 1) Cross Cluster Replication (CCR) refers to the process of replicating data from one cluster to another cluster, where a cluster includes at least one database, so cross cluster replication is also called cross database replication.

[0057] 2) Access log (Access.log) refers to the log file used to record access information.

[0058] 3) Version control, a technology for ensuring data consistency, is often used in data update or migration scenarios. In the embodiment of the present application, the update of the second modified data involves the implementation of version control technology.

[0059] 4) Incremental synchronization refers to the processing of differential data between the synchronization source database and the target database, also known as incremental update.

[0060] 5) Mirror backup, a data backup method; all data in the cluster can be backed up by creating a mirror of all data in the cluster. The database mirror in the embodiment of the present application is obtained by mirror backup.

[0061] 6) Gray process, a method of information release, refers to deploying the information to be released in a pre-release environment, and releasing the information to be released when the test of the information to be released passes; wherein the pre-release environment refers to an environment for testing the information to be released. In the embodiment of the present application, the gray process can be used to switch the database for business processing.

[0062] 7) Data consistency refers to the data consistency between different data versions; for example, the data consistency between the source database and the target database in the embodiment of the present application.

[0063] It should be noted that in the process of data processing, cross-database data migration is often involved. In order to migrate data across databases, CCR, reindexing and downtime migration can be used.

[0064] When using CCR to migrate data across databases, first, start a new database in a new hardware or cloud environment. Second, set up CCR in the source database, that is, add the new database information to the source database's configuration file (e.g., elasticsearch.yml) to set the new database as a follower database. Then, create a follower index in the new database for each index to be replicated in the source database. The follower index can be accessed through an interface (e.g., "PUT / <follower_index> / _ccr / follow" interface); then, start the replication and present the status of the replication process. Here, you can obtain the status of the replication process through the interface (for example, the "GET / _ccr / stats" interface). The replication time is negatively correlated with the amount of data and network bandwidth. Then, when the replication is completed, stop all write operations on the source database and send a response to the source database through the interface ("POST / <follower_index> / _ccr / pause_follow" interface) to stop the replication process; finally, convert the follower index to a normal index to perform write operations on the new database, which can be done through the interface (for example, "POST / <follower_index> / _ccr / unfollow" interface) converts the follower index into a normal index and points the application's database to the new database, thus completing the cross-database migration of data.

[0065] It should be noted that using CCR to migrate data across databases does not require downtime, and data can be replicated during business processing. However, some source databases (such as the community version of Elasticsearch database) cannot be migrated across databases through CCR, so using CCR to migrate across databases affects the adaptability of data migration.

[0066] When reindexing is used to migrate data across databases, first, set the source database information in the configuration file (elasticsearch.yml) in the new database (for example, "aaaaacodereindex:remote:src:hosts:["old_cluster:1000"]", where "src" is the alias of the source database and "old_cluster:1000" is the address of the source database). Second, perform the Reindex operation in the new database (you can use "Dev Tools" of "Kibana" or the "curl" command to execute an HTTP request, for example, "jsonCopycodePOST_reindex{"source":{"remote":{"host":"http: / / old_cluster:1000"},"index":"old_index"},"dest":{"index":"new_index"}}", where "old_index" refers to the index of the source database and "new_index" refers to the index of the new database). Then, after the Reindex operation is completed, verify whether the data in the index of the new database is complete and matches the data in the index of the source database. Finally, when the validation passes, point the application's database to the new source database.

[0067] It should be noted that rebuilding the index can move data from one database to another. However, the speed of data migration by rebuilding the data index is lower than the specified threshold, which affects the efficiency of data migration. In addition, the changes of incremental data cannot be synchronized during the data migration process, which affects the consistency of data.

[0068] For example, if 1,000 data items are migrated at a time and each operation takes 50 milliseconds, then migrating 100,000 data items takes 5 seconds, and migrating 1 billion data items takes 50,000 seconds. Figure 1 , Figure 1 is an exemplary migration time diagram; Figure 1 As shown, when re-indexing is used to perform full migration on the old database 1-1 (referred to as the source database), it takes 50,000 seconds, or half a day, to migrate 1 billion records; and it takes 500,000 seconds, or half a week, to migrate 10 billion records.

[0069] In addition, in order to improve the data consistency when reindexing is used to migrate data across databases, as shown in Table 1, four methods can be used: freezing the source index, synchronization mechanism, repeating the Reindex operation, and verifying the data.

[0070] Table 1

[0071]

[0072] It should be noted that freezing the source index will cause business interruption. Using a synchronization mechanism to perform dual write operations on the database will consume certain resources. Repeated reindex operations cannot achieve synchronous increments of deletion operations, and modification operations on the source database may overwrite newly modified data in the new database; see Figure 2 , Figure 2 is a schematic diagram that leads to dirty data; Figure 2 As shown in the figure, during the Reindex operation, data 2-11 in the old database 2-1 is deleted. Since the deletion operation cannot synchronize the increment, the deleted data 2-11 appears in the new database 2-2. Therefore, data 2-11 in the new database 2-2 becomes dirty data. Figure 3 , Figure 3 is a schematic diagram of an update coverage; Figure 3 As shown, the old updated data 3-11 in the old database 3-1 covers the new updated data 3-21 in the new database 3-2. The data verification method is a consistency verification technology that cannot solve the consistency problem. Therefore, the use of re-indexing to migrate data across databases affects the data migration efficiency and data consistency.

[0073] When using downtime migration to migrate data across clusters, first build a new database, where the version of the new database can be the same as or different from the source database. If the versions are different, the new database version can be compatible with the data of the old database version.

[0074] Secondly, stop all write operations on the source database. Then, create a data snapshot from the source database (for example, "bashCopy codePUT / _snapshot / my_backup / snapshot_1?wait_for_completion=true", where snapshot_1 is the name of the data snapshot), and restore the data snapshot on the new cluster (for example, "bashCopy codePOST / _snapshot / my_backup / snapshot_1 / _restore"). After that, check the integrity of the data on the new database to ensure that all data in the source database has been correctly migrated to the new database. Next, modify the application configuration to point the data read and write operations to the new database. Finally, start the write operation on the new database to restore the normal operation of the application.

[0075] It should be noted that during the process of cross-cluster data migration using downtime migration, all write operations are stopped, and smooth migration of the database for business processing cannot be achieved.

[0076] Based on this, the embodiments of the present application provide a data processing method, apparatus, device, computer-readable storage medium and computer program product, which can improve the smoothness, efficiency, scope of use and data consistency of data migration. The following describes an exemplary application of an electronic device (referred to as a data processing device) for data processing provided by the embodiments of the present application. The data processing device provided by the embodiments of the present application can be implemented as various types of terminals such as smart phones, smart watches, laptops, tablet computers, desktop computers, smart home appliances, set-top boxes, smart car devices, portable music players, personal digital assistants, dedicated messaging devices, intelligent voice interaction devices, portable game devices and smart speakers, and can also be implemented as a server, or as a combination of the two. Below, an exemplary application of the data processing device when it is implemented as a terminal will be described.

[0077] See also Figure 4 , Figure 4 Schematic diagram of the data processing system provided in the embodiment of the present application; Figure 4 As shown, to support a data processing application, in a data processing system 100, a terminal 400 is connected to a database 200 (referred to as a source database) and a database 500 (referred to as a target database) through a network 300, and a terminal 600 is connected to a server 700 through a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two. The database 200 and the database 500 are used to provide data support for the server 700; and, Figure 4 What is shown is a situation where the database 200 and the database 500 are independent of the server 700. In addition, the database 200 and the database 500 can also be integrated in the server 700, which is not limited in the embodiment of the present application.

[0078] Terminal 400 is used to migrate data in database 200 to database 500 through network 300: back up data in database 200 to obtain a database mirror; perform database recovery in database 500 based on the database mirror (graphic interface 410-1 of database recovery is exemplarily shown); update first changed data in database 200 to database 500, the first changed data being the data of database 200 changed during database recovery; switch the database for business processing from database 200 to database 500; in the process of performing business processing based on database 500, update second changed data in database 200 to database 500, the second changed data being the data of database 200 changed during the update of the first changed data; and continue to perform business processing based on database 500.

[0079] The terminal 600 is used to request services from the server 700 through the network 300 (exemplarily showing a graphical interface 610 - 1 for requesting information search services).

[0080] In some embodiments, server 700 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application. In addition, database 200 and database 700 can be a single database or a database cluster, which is not limited in the embodiments of the present application.

[0081] See also Figure 5 , Figure 5 This is a method provided by the embodiment of the present application. Figure 4 The schematic diagram of the terminal structure in FIG. Figure 5 As shown, the terminal 400 includes: at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 440 .

[0082] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0083] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0084] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0085] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0086] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0087] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0088] A network communication module 452, for reaching other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 include: Bluetooth, Wireless Fidelity (Wi-Fi), and Universal Serial Bus (USB), etc.;

[0089] a presentation module 453 for enabling presentation of information via one or more output devices 431 (e.g., display screen, speaker, etc.) associated with the user interface 430 (e.g., a user interface for operating peripherals and displaying content and information);

[0090] The input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0091] In some embodiments, the data processing device provided in the embodiments of the present application can be implemented in software. Figure 5The data processing device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an image acquisition module 4551, a data recovery module 4552, an incremental update module 4553, a business processing module 4554, a field addition module 4555 and a data determination module 4556. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0092] In some embodiments, the data processing device provided in the embodiments of the present application can be implemented in hardware. As an example, the data processing device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the data processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), DSP, programmable logic device (Programmable Logic Device, PLD), complex programmable logic device (Complex Programmable Logic Device, CPLD), field programmable gate array (Field-Programmable Gate Array, FPGA) or other electronic components.

[0093] In some embodiments, the terminal can implement the data processing method provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be microprogram-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a native (Native) application (APPlication, APP), that is, a program that needs to be installed in the operating system to run, such as a database operation and maintenance APP; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0094] The following will describe the data processing method provided by the embodiment of the present application in combination with the exemplary application and implementation of the data processing device provided by the embodiment of the present application. In addition, the data processing method provided by the embodiment of the present application is applied to various data migration scenarios such as cloud technology, artificial intelligence, smart transportation, information processing, maps, games, and vehicle-mounted.

[0095] See also Figure 6 , Figure 6This is a flow chart of the data processing method provided in the embodiment of the present application. Figure 1 , Figure 6 The execution subject of each step in the process is the data processing device; Figure 6 The steps shown are explained.

[0096] Step 101: Back up the source database to obtain a database image.

[0097] In the embodiment of the present application, when it is determined to perform data migration on the source database, the data processing device performs data backup on the source database, and the obtained data backup result is the database mirror.

[0098] It should be noted that the source database is the database to be migrated across databases; data backup refers to the process of backing up data through mirroring, so that the data processing device backs up the source database, that is, backs up the full amount of data in the source database at present (that is, the time when the data migration is initiated); therefore, the database mirror is a mirror of the full amount of data in the source database when the data migration is initiated. Among them, the data migration initiation time is the start time of the data backup. In addition, when the data migration is initiated (called the data migration initiation time), the database for business processing is the source database. In addition, business processing refers to data processing business, such as game business, information search business, and so on.

[0099] See also Figure 7 , Figure 7 This is a flow chart of the data processing method provided in the embodiment of the present application. Figure 2 , Figure 7 The execution subject of each step is the data processing device; in the embodiment of the present application, step 101 can be implemented through steps 1011 to 1013; that is, the data processing device backs up the data of the source database to obtain a database mirror, including steps 1011 to 1013, and each step is explained below.

[0100] Step 1011: Connect to the source database based on the source database address.

[0101] In the embodiment of the present application, since the source database is accessed through the source database address, when the data processing device creates a database mirror of the source database, it first establishes a connection with the source database through the source database address to access the source database. Here, the data processing device can access the source database based on the source database address through a database operation and maintenance tool.

[0102] It should be noted that the source database address is an access address of the source database, such as a Uniform Resource Locator (URL) of the source database; the source database address is used to access the source database.

[0103] Step 1012: Create a backup warehouse for the source database.

[0104] In the embodiment of the present application, the data processing device establishes a data warehouse for the source database to which the connection has been established, so as to store the data to be backed up in the source database; here, the established data warehouse is called a backup warehouse. The backup warehouse includes a backup warehouse name and a backup warehouse path address.

[0105] Step 1013: In the backup warehouse, a snapshot is created based on the index of the data to be backed up in the source database to obtain a database image.

[0106] In an embodiment of the present application, a data processing device creates a snapshot for a source database to complete the generation of a database image; here, the data processing device creates a snapshot by associating the index of the data to be backed up of the source database, thereby being able to back up the current full data in the source database to a backup warehouse; here, the snapshot created in the backup warehouse is the database image, and the index of the data to be backed up is the index of the current full data in the source database.

[0107] It is understandable that by backing up the full amount of data when initiating data migration in the source database through data backup, and completing the migration of the full amount of data when initiating data migration in the source database based on the backed-up database image, the efficiency of cross-database data migration can be improved.

[0108] Step 102: Perform database recovery in a designated database based on the database mirror to obtain an initial database.

[0109] In the embodiment of the present application, after the data processing device completes data backup of the source database when initiating data migration and obtains the database mirror, database recovery is performed in the new designated database based on the database mirror to achieve the recovery of the source database when initiating data migration. Here, the designated database that completes the database recovery is referred to as the initial database.

[0110] It should be noted that the designated database is a newly created database used to migrate the source database. The designated database can be a database with a higher version than the source database, a database with more functions than the source database, a database with higher performance than the source database, a database with a larger storage capacity than the source database, or a combination of the above, etc., which is not limited to the embodiments of the present application.

[0111] Continue to see Figure 7 In the embodiment of the present application, step 102 can be implemented through steps 1021 to 1023; that is, the data processing device performs database recovery in the specified database based on the database mirror to obtain the initial database, including steps 1021 to 1023, and each step is described below.

[0112] Step 1021: Connect to the specified database based on the specified database address.

[0113] In the embodiment of the present application, since the designated database is accessed through the designated database address, when the data processing device performs database recovery on the source database based on the database mirror in the designated database, it first establishes a connection with the designated database through the designated database address to access the designated database. Here, the data processing device can access the designated database based on the designated database address through a database operation and maintenance tool.

[0114] It should be noted that the designated database address is an access address of the designated database, for example, a URL of the designated database; the designated database address is used to access the designated database.

[0115] Step 1022: In the specified database, associate the backup repository of the source database.

[0116] In an embodiment of the present application, the data processing device associates the data source of the specified database as the backup warehouse of the source database through the backup warehouse name and the backup warehouse path address to obtain data from the backup warehouse and realize data recovery of the source database on the specified database.

[0117] Step 1023: Perform snapshot recovery on the database image in the backup warehouse to obtain the initial database.

[0118] In an embodiment of the present application, the data processing device performs a snapshot recovery on the database image in the associated backup warehouse in the specified database; when the snapshot recovery is completed, the migration of the full amount of data when the data migration is initiated in the source database is also completed in the specified database; thus, the initial database is equivalent to the source database at the beginning of the data backup.

[0119] Step 103: Update the first changed data in the source database to the initial database to obtain the database to be switched.

[0120] In an embodiment of the present application, during the time period corresponding to data backup and database recovery, the database for business processing is the source database, so that the changes to the source database during this time period are to be synchronized to the initial database; therefore, the data processing device obtains the data in the source database that has been changed between the start time of the data backup and the end time of the database recovery, and thus obtains the first changed data; and applies the first changed data to the initial database in an incremental update manner, and the initial database after completing the update of the first changed data is the database to be switched.

[0121] It should be noted that the first changed data is the data of the source database changed during the database recovery period, and the database recovery period refers to the time period between the start time of data backup and the end time of database recovery.

[0122] It can be understood that by updating the first change data during database recovery to the initial database, the changes in the source database during database recovery can be applied to the initial database, thereby improving the data integrity of the initial database and improving the data consistency between the initial database and the source database.

[0123] Continue to see Figure 7 In an embodiment of the present application, step 103 can be implemented through steps 1031 and 1032; that is, the data processing device updates the first changed data in the source database to the initial database to obtain the database to be switched, including steps 1031 and 1032, and each step is described below.

[0124] Step 1031: In the initial database, perform a first index migration on the first data to be updated in the source database to obtain a first intermediate database.

[0125] It should be noted that the first changed data includes the first data to be updated, and the first data to be updated is the data updated to the source database during database recovery, such as newly added data, modified data, a combination of the two, etc. Here, the data processing device updates the first data to be updated to the initial database by an index migration method of migrating data from one index to another index, and the initial database that completes the first index migration of the first data to be updated is called the first intermediate database. In addition, the first index migration refers to the index migration method performed on the first data to be updated.

[0126] In an embodiment of the present application, a data processing device performs a first index migration on the first data to be updated in a source database in an initial database to obtain a first intermediate database, including: the data processing device determines a first source data index of the first data to be updated in the source database; and determines a first target data index corresponding to the first source data index in the initial database; finally, performs a first index migration of data from the first source data index to the first target data index to obtain a first intermediate database.

[0127] It should be noted that, since the data processing device updates the first data to be updated to the initial database by index migration, the data index corresponding to the first data to be updated is determined in the source database, and the first source data index is obtained, and the data index corresponding to the first source data index is determined in the initial database, and the first target data index is obtained; here, the data processing device migrates the first data to be updated in the first source data index to the first target data index, and the first index migration is completed. Here, since there is a corresponding relationship between the data index in the source database and the data index in the initial database, and the first source data index is the data index in the source database, the data processing device can obtain the data index corresponding to the first source data index from the initial database, which is the first target data index.

[0128] Step 1032: Delete the first data to be deleted in the source database from the first intermediate database to obtain the database to be switched.

[0129] It should be noted that the first changed data also includes data deleted from the source database during database recovery, namely, the first data to be deleted. Here, after the data processing device determines the first data to be deleted, it deletes the first data to be deleted in the first intermediate database, and the first intermediate database after deleting the first data to be deleted is called the database to be switched.

[0130] It is understandable that after the first data to be updated is updated to the initial database to obtain the first intermediate database, the first data to be deleted that was deleted from the source database during database recovery is deleted from the first intermediate database, thereby improving the data consistency between the source database and the first intermediate database.

[0131] In an embodiment of the present application, a data processing device deletes the first data to be deleted in a source database from a first intermediate database to obtain a database to be switched, including: the data processing device first obtains a target access log of the source database during database recovery; and parses the target deletion operation in the target access log; finally, executes the target deletion operation in the first intermediate database to obtain the database to be switched.

[0132] It should be noted that the access log is used to record the operation records of the source database, and the access log carries the access time, so that the data processing device can obtain the operation records of the source database during the database recovery period, and the operation records of the source database during the database recovery period are referred to as the target access log; here, the data processing device parses the target access log to parse the deletion operation in the target access log, and the parsed deletion operation is referred to as the target deletion operation; in addition, the target deletion operation is used to delete the first data to be deleted, and the data processing device performs the target deletion operation in the first intermediate database to delete the first data to be deleted from the first intermediate database, so that the deleted first intermediate database is consistent with the data in the source database. Among them, the first intermediate database after the deletion of the first data to be deleted is the database to be switched.

[0133] In an embodiment of the present application, the data processing device may also capture the operation of the source database by operation capture to delete the first data to be deleted from the first intermediate database, which is not limited in the embodiment of the present application.

[0134] Step 104: Switch the database for business processing from the source database to the database to be switched.

[0135] It should be noted that, since the database to be switched includes not only the full data before the start time of the data backup, but also the first changed data of the source database during the database recovery, the database to be switched can replace the source database for business processing; therefore, the data processing device switches the database for business processing from the source database to the database to be switched, and stops business processing on the source database.

[0136] It is understandable that before switching business processing to a new database, the data in the source database is migrated to the database to be switched by combining data backup and update methods, thereby achieving smooth migration of business processing and improving data consistency between the database to be switched and the source database.

[0137] Step 105 : In the process of performing business processing based on the database to be switched, the second changed data in the source database is updated to the database to be switched to obtain a target database.

[0138] In an embodiment of the present application, after completing the database switching, in the process of performing business processing based on the database to be switched, the data processing device updates the second change data in the source database to the database to be switched, so as to apply the change data during the first incremental update in the source database to the switching database; wherein the second change data is the data changed to the source database during the first incremental update, that is, the data changed to the source database during the update of the first change data; in addition, the database to be switched that has completed the second change data update is the target database.

[0139] It can be understood that the second change data is the data of the source database changed during the update of the first change data; since there are changes to the source database during the update of the first change data, the data processing device performs another update after completing the database switch to improve the data integrity of the target database and also improve the data consistency between the target database and the source database.

[0140] Continue to see Figure 7 In an embodiment of the present application, step 105 can be implemented through steps 1051 and 1052; that is, the data processing device updates the second changed data in the source database to the database to be switched to obtain the target database, including steps 1051 and 1052, and each step is described below.

[0141] Step 1051: In the database to be switched, perform a second index migration on the second data to be updated in the source database to obtain a second intermediate database.

[0142] It should be noted that the second changed data includes the data updated on the source database during the update of the first changed data, that is, the second data to be updated, such as newly added data, modified data, a combination of the two, etc. Here, the data processing device updates the second data to be updated to the database to be switched by index migration, and the database to be switched that completes the second index migration of the second data to be updated is called the second intermediate database. In addition, the second index migration refers to the index migration method performed on the second data to be updated.

[0143] In an embodiment of the present application, a data processing device performs a second index migration on second data to be updated in a source database in a database to be switched to obtain a second intermediate database, including: the data processing device determines a second source data index of the second data to be updated in the source database; and determines a second target data index corresponding to the second source data index in the database to be switched; finally, performs a second index migration of data from the second source data index to the second target data index to obtain a second intermediate database.

[0144] It should be noted that, since the data processing device uses the index migration method to update the second data to be updated to the database to be switched, the data index corresponding to the second data to be updated is determined in the source database, and the second source data index is obtained, and the data index corresponding to the second source data index is determined in the database to be switched, and the second target data index is obtained; here, the data processing device migrates the second data to be updated in the second source data index to the second target data index, and the second index migration is completed. Here, since there is a corresponding relationship between the data index in the source database and the data index in the database to be switched, and the second source data index is the data index in the source database, the data processing device can obtain the data index corresponding to the second source data index from the database to be switched, which is the second target data index.

[0145] In an embodiment of the present application, a data processing device performs a second index migration on second data to be updated in a source database in a database to be switched to obtain a second intermediate database, including: the data processing device first determines, in the database to be switched, a target data version corresponding to the second data to be updated in the source database; then, in the source database, determines a source data version corresponding to the second data to be updated; then, from the second data to be updated, filters target data to be updated whose target data version is smaller than the source data version; finally, in the database to be switched, performs a second index migration on the target data to be updated to obtain a second intermediate database.

[0146] It should be noted that, since the database for business processing is the database to be switched when the data processing device executes the second index migration, there is a problem of overwriting the latest modified data in the database to be switched if the second index migration is executed directly; therefore, the data processing device adopts version control to solve the data coverage problem. Since the data version is positively correlated with the latest degree of data update, by comparing the data versions, it is possible to determine whether there is data coverage; and when the data version in the database to be switched is greater than or equal to the corresponding data version in the source database, it indicates that the data in the database to be switched is the latest data, and the process of migrating the corresponding data in the source database to the database to be switched is canceled; and when the data version in the database to be switched is less than the corresponding data version in the source database, it indicates that the data in the database to be switched is to be updated; thus, the data processing device selects the target data to be updated whose target data version is less than the source data version from the second data to be updated, and migrates the target data to be updated to the database to be switched. In addition, the source data version is the data version in the source database corresponding to the second data to be updated, and the target data version is the data version in the database to be switched corresponding to the second data to be updated.

[0147] It is understandable that, in the process of executing the second index migration, the data processing device controls the target data to be migrated and updated through data version control, thereby improving the accuracy of data migration and further improving the data integrity and correctness of the second intermediate database.

[0148] Step 1052: Delete the second data to be deleted in the source database from the second intermediate database to obtain the target database.

[0149] It should be noted that the second changed data includes data deleted from the source database during the update of the first changed data, that is, the second data to be deleted. Here, after the data processing device determines the second data to be deleted, it deletes the second data to be deleted in the second intermediate database, and the second intermediate database after deleting the second data to be deleted is called the target database.

[0150] It can be understood that after the second data to be updated is updated to the database to be switched to obtain the second intermediate database, the second data to be deleted that was deleted from the source database during the update of the first changed data is deleted from the second intermediate database, thereby improving the database consistency between the source database and the second intermediate database.

[0151] In an embodiment of the present application, a data processing device deletes the second data to be deleted in a source database from a second intermediate database to obtain a target database, including: the data processing device first obtains an access log of the source database during the update period of the first change data; and parses the deletion operation in the access log during the update period of the first change data; finally, the parsed deletion operation is executed in the second intermediate database, thereby obtaining the target database. Here, the parsed deletion operation is used to delete the second data to be deleted.

[0152] In an embodiment of the present application, the data processing device may further synchronize the first data to be deleted and the second data to be deleted in the source database after executing the second index migration, which is not limited in the embodiment of the present application.

[0153] Step 106: Perform business processing based on the target database.

[0154] It should be noted that after the data processing device obtains the target database, it continues to use the target database as a database for business processing.

[0155] It is understandable that when performing data migration across databases on the source database, the source database is first restored in the specified database through the data backup method of database mirroring; then the database is updated before and after the database switch to complete the switch of the business processing database from the source database to the target database; because the database recovery speed of the data backup is greater than the update speed, the database is first restored through the data backup method to complete the migration of the full amount of data in the source data; then the update is performed to complete the data update of the source data after the data backup; this can improve the efficiency of data migration. In addition, it can also achieve smooth migration across databases, ensuring the stability of business processing.

[0156] In an embodiment of the present application, before the data processing device backs up data in the source database to obtain a database mirror, the data processing method further includes: the data processing device adds a latest update time field in the source database, and the latest update time field is used to record the latest update time of the data; thereby, the data processing device updates the first changed data in the source database to the initial database, and before obtaining the database to be switched, the data processing method further includes: the data processing device determines, in the source database, the data whose latest update time is the data changed during database recovery as the first changed data; and the data processing device updates the second changed data in the source database to the database to be switched, and before obtaining the target database, the data processing method further includes: the data processing device determines, in the source database, the data whose latest update time is the data changed during the update of the first changed data as the second changed data.

[0157] In an embodiment of the present application, before the data processing device switches the database for business processing from the source database to the database to be switched, the data processing method also includes: the data processing device determines the initial database as the database for verifying the business processing; thereby, the data processing device switches the database for business processing from the source database to the database to be switched, including: when it is determined that the verification business processing has passed in the process of verifying the business processing based on the initial database, the database for business processing is switched from the source database to the database to be switched.

[0158] It should be noted that the verification service process passes, which means that the error rate when executing the verification service process based on the database to be switched is lower than the first specified threshold.

[0159] In an embodiment of the present application, after the data processing device performs business processing based on the target database, the data processing method further includes: when the data processing device determines that the business processing is passed during the business processing based on the target database, the source database is deleted. In this way, resource consumption can be reduced. Wherein, the business processing is passed, indicating that the error rate when executing the business processing based on the target database is lower than the second specified threshold.

[0160] In the embodiment of the present application, the business processing may be related processing in the field of information search. At this time, the migration data includes account data, business information data, information interaction data, and the like.

[0161] Below, an exemplary application of the embodiment of the present application in an actual application scenario will be described. The exemplary application describes the process of performing cross-database data migration in a search system. Among them, the search system is implemented based on a database (for example, an Elasticsearch database) to realize the retrieval of data such as information, dynamics, accounts, communities, and interactive messages. When the data changes, the search system synchronizes a copy of the corresponding data to the database.

[0162] For example, see Figure 8 , Figure 8 is an exemplary application schematic diagram provided in the embodiment of the present application; Figure 8 As shown, interface 8-1 is a comprehensive data retrieval interface, which displays the retrieved comprehensive data.

[0163] See also Fig. 9 , Fig. 9 is another exemplary application schematic diagram provided in the embodiment of the present application; Fig. 9 As shown, interface 9-1 is an account data retrieval interface, which displays the retrieved account data.

[0164] It should be noted that when the source database version needs to be updated or the load value is greater than the load threshold, for example, when the database version on which the game is based needs to be updated, or the load value of the database on which the game is based reaches 60-80%, the search frequency (300 times / second) is close to (the difference with the upper limit is lower than the difference threshold) the upper limit (625 times / second), or the indexing frequency (750 times / second) is greater than the upper limit (670 times / second), etc., the database version is often updated or the load value is reduced through cross-database data migration.

[0165] See also Fig.10 , Fig.10 is an exemplary load schematic diagram provided in an embodiment of the present application; Fig.10 As shown, in the time period 10-1, the CPU load value 10-3 corresponding to each shard 10-2 has exceeded 60% at most. At this time, the load value can be reduced by cross-database data migration.

[0166] It should be noted that the embodiment of the present application combines the mirror backup and incremental data reindexing methods to perform data migration. First, the data of the source database is packaged and backed up using the database operation and maintenance tool, and then a new database is rebuilt through physical backup and recovery. Then, the query function in the database's rekey indexing function is used to synchronize data whose latest update time is greater than the specified time (i.e., the mirror backup start time) to achieve data migration of the rekey index of incremental data. At this point, see Fig.11 , Fig.11 is a schematic diagram of a data migration method provided in an embodiment of the present application; Fig.11 As shown, for the data in the old database (called the source database), mirror backup 11-1 (including data backup and database recovery) and incremental method 11-2 (including the update of the first changed data and the second changed data) are used for migration. The backup and recovery of 10 billion data takes 2 hours, and the incremental migration of 100,000 data takes several minutes.

[0167] The overall structure of this application is described below.

[0168] See also Fig.12 , Fig.12 is an exemplary overall architecture diagram provided in the embodiment of the present application; Fig.12 As shown, the overall architecture 12-1 includes an old database 12-11, a new database 12-12, an operation and maintenance tool 12-13 (for example, Cerebro) and a business service 12-14. Among them, the operation and maintenance tool 12-13 is used to package and back up the data, and then rebuild a new database (for example, a new Elasticsearch database) in the new business environment through physical backup and recovery. After the data is backed up and the database is restored, the incremental data between the backup start time and the database recovery end time is incrementally updated; here, the incremental update is achieved by performing the Reindex operation; during the incremental update process, it is also used to synchronize the deletion operations to ensure the consistency of the new and old databases. Here, a data deletion synchronization script is used to parse the deletion operations recorded in the access log of the application layer, and after the Reindex operation, the parsed deletion operations are replayed in the new database.

[0169] Continue to see Fig.12The smooth migration process of the database includes data backup (step 1201), database recovery (step 1202), the first incremental data processing (step 1203), the first synchronous deletion operation (step 1204), database switching (step 1205), the second incremental data processing (step 1206) and the second synchronous deletion operation (step 1207). Data backup refers to using the operation and maintenance tool 12-13 to back up the data in the old database 12-11; database recovery refers to using the backed-up data to rebuild the database to obtain a new database 12-12; database switching refers to pointing the database of the business service 12-14 from the old database 12-11 to the new database 12-12; the first incremental data processing refers to migrating the data newly written to the old database 12-11 during data backup and database recovery to the new database 12-12, and the second incremental data processing refers to migrating the data newly written to the old database 12-11 during the first incremental data processing and the first synchronous deletion operation to the new database 12-12; the first synchronous deletion operation and the second synchronous deletion operation refer to using the data deletion synchronization script 12-15 to parse the deletion operations of the corresponding period in the application layer access log, and execute the parsed deletion operations in the new database 12-12 after the incremental data processing is completed. In addition, during the data migration process, a version control mechanism is used to ensure data consistency; the version control mechanism means that when performing the Reindex operation, the version in the source database is copied to the new database, and the data in the new database whose version is lower than the version in the source database is updated.

[0170] Below Fig.12 The processes from step 1201 to step 1207 are described separately.

[0171] Step 1201: Data backup.

[0172] It should be noted that the operation and maintenance tools 12-13 are management tools for the database, and are used to create and restore snapshots. When performing data backup, first, the operation and maintenance tool login is completed.

[0173] Here, you can access the management interface of the operation and maintenance tool by searching the URL of the operation and maintenance tool in the browser; log in to the account on the management interface to complete the login of the maintenance tool.

[0174] Next, connect to the source database.

[0175] Here, in the management interface after login, in response to the input operation of the URL of the source database, the URL of the source database is obtained; in response to the operation of clicking "connect" for the URL of the source database, the source database is connected to.

[0176] Then, create a backup repository.

[0177] Here, the "Create Warehouse" control is presented in the interface of the operation and maintenance tool. By responding to the operation that triggers the "Create Warehouse" control, the input backup warehouse and backup warehouse path are obtained. The corresponding program is "bash PUT / _snapshot / my_backup{"type":"fs","settings":{"location":" / mount / backups / my_backup"}}", where my_backup refers to the name of the backup warehouse and / mount / backups / my_backup refers to the path of the backup warehouse.

[0178] Finally, create a snapshot in the backup repository.

[0179] Here, a "Create Snapshot" control is presented in the interface of the operation and maintenance tool. By responding to the operation of triggering the "Create Snapshot" control, the input snapshot and the index to be backed up selected from the source database are obtained. The program for obtaining the input snapshot is "bash PUT / _snapshot / my_backup / snapshot_1?wait_for_completion=true", where snapshot_1 refers to the name of the snapshot, and wait_for_completion=true means waiting for the snapshot to be created.

[0180] It should be noted that, at this point, the data backup process of the source database using the operation and maintenance tool has been completed. The next step is to restore the snapshot in the new environment to achieve database recovery.

[0181] Step 1202: Database recovery.

[0182] It should be noted that database recovery refers to the process of restoring the source database in a new database using backup data (called database mirroring). First, connect to the new database (called the designated database).

[0183] Here, in the management interface after login, in response to the input operation of the URL of the new database, the URL of the new database is obtained; in response to the operation of clicking "Connect" for the URL of the new database, the connection to the new database is made.

[0184] Next, associate the backup repository.

[0185] Here, for the new database, the operation and maintenance tool interface presents a "Create Warehouse" control. By responding to the operation that triggers the "Create Warehouse" control, the selected backup warehouse and backup warehouse path are obtained. The corresponding program is "bashPUT / _snapshot / my_backup{"type":"fs","settings":{"location":" / mount / backups / my_backup"}}", where the backup warehouse path / mount / backups / my_backup is the path of the backup data that the new database can access.

[0186] Finally, restore the snapshot.

[0187] Here, a "Restore Snapshot" control is presented in the interface of the operation and maintenance tool, and the selected created snapshot and the index to be backed up are obtained by responding to the operation of triggering the "Restore Snapshot" control. The program for restoring the snapshot is "bash POST / _snapshot / my_backup / snapshot_1 / _restore".

[0188] It should be noted that the rate of database recovery is related to the size of the snapshot and the network conditions. After the recovery is completed, the new database contains all the backup data, and then data increment processing can be performed.

[0189] Step 1203: First incremental data processing.

[0190] It should be noted that the first incremental data processing refers to updating the update information of the old database in the period between the start time of data backup and the end time of database recovery to the new database. Here, the Reindex operation is used to implement the first incremental data processing.

[0191] Step 1204: The first synchronous deletion operation.

[0192] It should be noted that, when the first incremental data processing is performed, the deletion operations in the corresponding period cannot be applied to the new database through incremental updates; therefore, after the first incremental data processing is completed, the access log corresponding to the time period between the data backup start time and the database recovery end time is parsed through the data deletion synchronization script to execute the parsed deletion operations in the new database.

[0193] Step 1205: Database switching.

[0194] It should be noted that in order to reduce the impact of database switching, a grayscale process is used to switch the database to ensure smooth transition and continuity of business. First, switch to the new database in the pre-release environment. Then, when the new database is confirmed to be available in the pre-release environment, switch to the new database in the official environment.

[0195] It is understandable that the new database is verified in the pre-release environment to verify the stability and reliability of the new database, and the new database is switched to the official environment only after the verification is passed. In other words, switching the database through the grayscale process can minimize the impact of database migration while ensuring business continuity and ensure business stability.

[0196] Step 1206: Second incremental data processing.

[0197] It should be noted that after completing the switch of the new database, since there is incremental data updated on the source database during the execution of the first incremental data processing and the first synchronous deletion operation, the Reindex operation is used to apply the incremental data during the first incremental data processing and the first synchronous deletion operation in the source database to the new database. Since at this time, the database for business processing is a new database, version control is performed when performing the second incremental data processing to ensure data consistency and integrity. First, select the source database and the new database in the Reindex interface; then, through range search (Range Query), and use the first incremental data processing and the first synchronous deletion operation as the time condition to find new or modified data in the source database; then, add a version parameter (version_type) in the new database (dest) and set it to allow replication (external) to copy the version field (version) of the index in the source database to the index of the new database; finally, by specifying the source database and the index name of the source database, read the data corresponding to the index name from the source database (source).

[0198] The program corresponding to the incremental data processing is "POST_reindex{"source":{"remote":{"host":"http: / / otherhost:1000","username":"user","password":"pass"},"index":"source-index","query":{"range":{"updateAt":{"gt":"_datetime"}}}},"dest":{"index":"dest-index","version_type":"external"},"throttle":"requests_per_second"}"; otherhost:1000 is the address of the source database, source-index represents the index of the source database, and _datetime is the time of the first incremental data processing of the source database. In the new database part, dest-index represents the index of the new database; requests_per_second is the number of requests allowed per second, which can be adjusted according to actual needs, and can be set to an integer, a floating point number, or -1 (indicating that the number of requests is not limited), etc.

[0199] Step 1207: Second synchronous deletion operation.

[0200] It should be noted that, when executing the second incremental data processing, the deletion operation of the corresponding period cannot be applied to the new database through incremental update; therefore, after executing the second incremental data processing, the access log of the corresponding period of the application layer is parsed by the data deletion synchronization script to obtain the corresponding deletion operation; and when the deletion operation is detected, the deletion operation is recorded; and after executing the second incremental data processing, the recorded deletion operation is replayed in the new database to realize the synchronous deletion of data in the new database.

[0201] Among them, the program for synchronous deletion operation includes a program for parsing logs, namely "delete_operations=[]for line in access.log:if'DELETE'in line:", a program for parsing deletion operation, namely "delete_operation=parse_delete_operation(line)delete_operations.append(delete_operation)", and a program for replaying deletion operation, namely "for delete_operation indelete_operations:replay_delete_operation(delete_operation)"; here, the parse_delete_operation function is used to parse detailed information of the deletion operation from the line of the access log, such as the identification (ID) of the deleted document, and the replay_delete_operation function is used to execute the parsed deletion operation in the new database.

[0202] It should be noted that the program used for the data processing method described in the embodiment of the present application may be a computer executable program or a pseudo code, and the embodiment of the present application does not limit this.

[0203] In the embodiment of the present application, the processing flow of the first incremental data processing and the second incremental data processing is similar, except that the time period is different. The processing flow of the first synchronous deletion operation and the second synchronous deletion operation is similar, except that the corresponding time period is different; and the second incremental data processing adopts version control to solve the problem of old updated data in the source database overwriting new updated data in the new database. Here, the first synchronous deletion operation can also be canceled, and after completing the second incremental data processing, the deletion operation in the source database after the data backup start time is synchronized.

[0204] It can be understood that by performing the deletion operation of the corresponding period after the incremental data processing, it can be ensured that during the migration process, all deletion operations in the source database can be correctly applied to the new database, thereby ensuring the consistency of the new and old databases.

[0205] See also Fig.13 , Fig.13 is another exemplary load schematic diagram provided in an embodiment of the present application; Fig.13As shown, when the cross-database data migration is completed, in the time period 13-1 of applying the new database to perform business processing, the maximum and current load values ​​13-3 corresponding to each shard 13-2 are less than 15%. The card 13-4 also exemplarily shows the load values ​​of each shard 13-2 at a specified time (…-05-08 21:59:00).

[0206] The data migration cost provided in the embodiment of the present application is analyzed below.

[0207] See Table 2, which describes the comparison results of baseline solution 1, baseline solution 2 and the solution provided by the embodiment of the present application (hereinafter referred to as the solution of the present application) in terms of storage cost, event cost, database migration time and resource consumption.

[0208] Table 2

[0209]

[0210] It is understandable that the embodiment of the present application realizes cross-database data migration while ensuring the normal operation of business processing, provides continuous data storage and query services during the migration process, and improves the continuity and stability of business processing. In addition, during the data migration process, the timestamp field updateAt is introduced, so that incremental update processing can be performed after the database is restored, and then the mirrored data backup and incremental update method can be adopted to simplify the data migration process, reduce data migration consumption, and improve data migration efficiency and data consistency. In addition, after the incremental update, the access log of the application layer is parsed through the data deletion synchronization script, so that the deletion operation during the incremental update can be applied to the new database, thereby improving the data consistency and integrity between databases. Moreover, in the process of incremental data update after completing the database switch, the use of version control can improve the accuracy of incremental data update, thereby improving the data consistency and integrity between databases.

[0211] The following is a description of an exemplary structure of the data processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 5 As shown, the software modules stored in the data processing device 455 of the memory 450 may include:

[0212] The image acquisition module 4551 is used to back up the source database and obtain a database image;

[0213] A data recovery module 4552 is used to perform database recovery in a specified database based on the database mirror to obtain an initial database;

[0214] The incremental update module 4553 is used to update the first changed data in the source database to the initial database to obtain the database to be switched, wherein the first changed data is the data changed during database recovery;

[0215] A business processing module 4554, configured to switch a database for business processing from the source database to the database to be switched;

[0216] The incremental update module 4553 is further used to update the second changed data in the source database to the database to be switched to obtain a target database during business processing based on the database to be switched, wherein the second changed data is data changed during the update of the first changed data;

[0217] The business processing module 4554 is also used to perform business processing based on the target database.

[0218] In an embodiment of the present application, the incremental update module 4553 is also used to perform a first index migration on the first data to be updated in the source database in the initial database to obtain a first intermediate database, wherein the first change data includes the first data to be updated updated during the recovery of the database; and delete the first data to be deleted in the source database from the first intermediate database to obtain the database to be switched, wherein the first change data also includes the first data to be deleted deleted during the recovery of the database.

[0219] In an embodiment of the present application, the incremental update module 4553 is also used to determine the first source data index of the first data to be updated in the source database; determine the first target data index corresponding to the first source data index in the initial database; and execute the first index migration of data from the first source data index to the first target data index to obtain the first intermediate database.

[0220] In an embodiment of the present application, the incremental update module 4553 is also used to obtain the target access log of the source database during the database recovery period; parse the target deletion operation in the target access log, and the target deletion operation is used to delete the first data to be deleted; execute the target deletion operation in the first intermediate database to obtain the database to be switched.

[0221] In the embodiment of the present application, the data processing device 455 further includes a field adding module 4555, which is used to add a latest update time field in the source database, and the latest update time field is used to record the latest update time of the data.

[0222] In the embodiment of the present application, the data processing device 455 further includes a data determination module 4556, which is used to determine, in the source database, the data whose latest update time is changed during the database recovery as the first changed data.

[0223] In the embodiment of the present application, the data determination module 4556 is further used to determine, in the source database, data whose latest update time is changed during the update period of the first changed data as the second changed data.

[0224] In an embodiment of the present application, the incremental update module 4553 is also used to perform a second index migration on the second data to be updated in the source database in the database to be switched, to obtain a second intermediate database, wherein the second change data includes the second data to be updated updated during the update of the first change data; and to delete the second data to be deleted in the source database from the second intermediate database, to obtain the target database, wherein the second change data includes the second data to be deleted deleted during the update of the first change data.

[0225] In an embodiment of the present application, the incremental update module 4553 is also used to determine, in the database to be switched, the target data version corresponding to the second data to be updated in the source database; determine, in the source database, the source data version corresponding to the second data to be updated; filter, from the second data to be updated, the target data to be updated whose target data version is smaller than the source data version; and, in the database to be switched, perform a second index migration on the target data to be updated to obtain the second intermediate database.

[0226] In an embodiment of the present application, the image acquisition module 4551 is also used to connect to the source database based on the source database address, and the source database address is used to access the source database; create a backup warehouse for the source database; in the backup warehouse, create a snapshot based on the index of the data to be backed up in the source database to obtain the database image.

[0227] In an embodiment of the present application, the data recovery module 4552 is also used to connect to the specified database based on a specified database address, and the specified database address is used to access the specified database; in the specified database, the backup warehouse of the source database is associated; and the database image in the backup warehouse is snapshot restored to obtain the initial database.

[0228] In the embodiment of the present application, the business processing module 4554 is further used to determine the initial database as a database for verifying business processing.

[0229] In the embodiment of the present application, the business processing module 4554 is also used to switch the database of the business processing from the source database to the database to be switched when it is determined that the verification business processing passes during the process of verifying the business processing based on the initial database.

[0230] In the embodiment of the present application, the business processing module 4554 is further used to delete the source database when it is determined that the business processing is passed during the business processing based on the target database.

[0231] The embodiment of the present application provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer-readable storage medium. The processor of the data processing device reads the computer executable instructions or computer programs from the computer-readable storage medium, and the processor executes the computer executable instructions or computer programs, so that the data processing device performs the data processing method described above in the embodiment of the present application.

[0232] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the data processing method provided by the present application embodiment, for example, Figure 6 The data processing method is shown.

[0233] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0234] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0235] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0236] As an example, computer-executable instructions may be deployed to be executed on one electronic device (in which case, this one electronic device is a data processing device), or on multiple electronic devices located at one location (in which case, multiple electronic devices located at one location are data processing devices), or on multiple electronic devices distributed at multiple locations and interconnected by a communication network (in which case, multiple electronic devices distributed at multiple locations and interconnected by a communication network are data processing devices).

[0237] It is understandable that in the embodiments of the present application, related data such as data and addresses in the database are involved. When the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0238] In summary, the embodiment of the present application, when performing data migration across databases on the source database, first restores the source database in the specified database through the data backup method of database mirroring; then performs incremental updates before and after the database switch to complete the switching of the business processing database from the source database to the target database; since the database recovery speed of the data backup is greater than the incremental update speed, the database is first restored through the data backup method to complete the migration of the full amount of data in the source data; then the incremental update is performed to complete the data update of the source data after the data backup; the data migration efficiency can be improved. In addition, smooth migration across databases can be achieved to ensure the stability and continuity of business processing. In addition, when performing incremental updates, the update data is first updated by rebuilding the index, and then the deleted data is synchronized, which can improve the data consistency between databases; and when performing incremental updates after completing the database switch for business processing, the accurate synchronization of the updated data is achieved by using version control, which can improve the data consistency between databases.

[0239] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A data processing method, characterized in that: The method comprises: Back up the source database to obtain a database image; Performing database recovery in a designated database based on the database mirror to obtain an initial database; Updating the first changed data in the source database to the initial database to obtain the database to be switched, wherein the first changed data is the data changed during database recovery; Switching the database for business processing from the source database to the database to be switched; In the process of performing business processing based on the database to be switched, the second changed data in the source database is updated to the database to be switched to obtain a target database, wherein the second changed data is data changed during the update of the first changed data; Business processing is performed based on the target database.

2. The method according to claim 1, characterized in that The updating of the first changed data in the source database to the initial database to obtain the database to be switched includes: In the initial database, performing a first index migration on the first data to be updated in the source database to obtain a first intermediate database, wherein the first changed data includes the first data to be updated updated during the database recovery period; The first data to be deleted in the source database is deleted from the first intermediate database to obtain the database to be switched, and the first changed data further includes the first data to be deleted deleted during the recovery of the database.

3. The method according to claim 2, characterized in that The performing of a first index migration on the first data to be updated in the source database in the initial database to obtain a first intermediate database includes: In the source database, determining a first source data index of the first data to be updated; In the initial database, determining a first target data index corresponding to the first source data index; A first index migration is performed to migrate data from the first source data index to the first target data index to obtain the first intermediate database.

4. The method according to claim 2, characterized in that: The deleting the first data to be deleted in the source database from the first intermediate database to obtain the database to be switched includes: Obtain a target access log of the source database during the database recovery period; parsing a target deletion operation in the target access log, where the target deletion operation is used to delete the first data to be deleted; The target deletion operation is performed in the first intermediate database to obtain the database to be switched.

5. The method according to any one of claims 1 to 4, characterized in that: Before backing up the source database to obtain the database mirror, the method further includes: Adding a latest update time field in the source database, wherein the latest update time field is used to record the latest update time of the data; Before updating the first changed data in the source database to the initial database to obtain the database to be switched, the method further includes: In the source database, determining the data whose latest update time is changed during the database recovery period as the first changed data; Before updating the second changed data in the source database to the database to be switched to obtain the target database, the method further includes: In the source database, data whose latest update time is changed during the update period of the first changed data is determined as the second changed data.

6. The method according to any one of claims 1 to 4, characterized in that: The updating of the second changed data in the source database to the database to be switched to obtain the target database comprises: In the database to be switched, performing a second index migration on the second data to be updated in the source database to obtain a second intermediate database, wherein the second changed data includes the second data to be updated updated during the update of the first changed data; The second data to be deleted in the source database is deleted from the second intermediate database to obtain the target database, wherein the second changed data includes the second data to be deleted deleted during the update of the first changed data.

7. The method according to claim 6, characterized in that The step of performing a second index migration on the second data to be updated in the source database in the database to be switched to obtain a second intermediate database includes: In the database to be switched, determining a target data version corresponding to the second data to be updated in the source database; Determining, in the source database, a source data version corresponding to the second data to be updated; Filter the target data to be updated whose target data version is smaller than the source data version from the second data to be updated; In the database to be switched, a second index migration is performed on the target data to be updated to obtain the second intermediate database.

8. The method according to any one of claims 1 to 4, characterized in that: The step of backing up the source database to obtain a database mirror includes: Connecting to the source database based on a source database address, wherein the source database address is used to access the source database; Creating a backup warehouse for the source database; In the backup warehouse, a snapshot is created based on the index of the data to be backed up in the source database to obtain the database image.

9. The method according to any one of claims 1 to 4, characterized in that: The step of performing database recovery in a designated database based on the database mirror to obtain an initial database includes: Connecting to the specified database based on a specified database address, wherein the specified database address is used to access the specified database; In the designated database, associating a backup repository of the source database; Perform snapshot recovery on the database image in the backup warehouse to obtain the initial database.

10. The method according to any one of claims 1 to 4, characterized in that: Before switching the database for business processing from the source database to the database to be switched, the method further includes: Determining the initial database as a database for verifying business processing; The step of switching the database for business processing from the source database to the database to be switched includes: When it is determined that the verification business process has passed during the process of performing the verification business process based on the initial database, the database of the business process is switched from the source database to the database to be switched.

11. The method according to any one of claims 1 to 4, characterized in that: After performing business processing based on the target database, the method further includes: When it is determined that the business processing is successful during the business processing based on the target database, the source database is deleted.

12. A data processing device, characterized in that: The data processing device comprises: The image acquisition module is used to back up the source database and obtain the database image; A data recovery module, used for performing database recovery in a designated database based on the database mirror to obtain an initial database; An incremental update module, configured to update first changed data in the source database to the initial database to obtain a database to be switched, wherein the first changed data is data changed during database recovery; A business processing module, used to switch a database for business processing from the source database to the database to be switched; The incremental update module is further used to update the second changed data in the source database to the database to be switched to obtain a target database during business processing based on the database to be switched, wherein the second changed data is data changed during the update of the first changed data; The business processing module is also used to perform business processing based on the target database.

13. An electronic device for data processing, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, configured to implement the data processing method according to any one of claims 1 to 11 when executing the computer executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that: When the computer executable instructions or computer programs are executed by a processor, the data processing method according to any one of claims 1 to 11 is implemented.

Images