Data migration detection method and device, equipment, storage medium and program product

Abstract

The invention relates to the field of data processing, and particularly discloses a data migration detection method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining a first data migration result set and a second data migration result set in a data migration process; obtaining a third data migration result set in the data migration process, wherein the third data migration result set comprises all migration data of the emigrated host stored in the distributed database after data migration is completed; combining the first data transfer result set and the second data transfer result set to obtain a detection set; traversing each piece of data in the third data migration result set, comparing the data with the detection set, and determining a data migration detection result; according to the data migration detection result, target indication information is generated, and the target indication information is used for indicating the distributed database to delete redundant data in the migration data stored in the migration host. By adopting the method, the time consumption of data detection after data migration can be reduced.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data migration detection method, device, equipment, storage medium and program product. Background Art

[0002] At present, all institutions use the mainframe to support the operation of their core businesses. The highly integrated data system runs on the mainframe based on the centralized architecture. Its software and hardware product technologies are relatively mature and can provide stable, reliable and high-quality services for the core businesses of various institutions. In order to ensure the security and stability of the data system and get rid of the dependence on the single technology of the mainframe, various institutions usually have the need to migrate data between different mainframes.

[0003] In the related art, data migration between different hosts is performed in a shutdown state. After completing the data migration in the shutdown state, each migrated data needs to be tested one by one. When the amount of migrated data is large, it will take a long time to test the migrated data. Summary of the invention

[0004] Based on this, it is necessary to provide a data migration detection method, device, equipment, storage medium and program product that can reduce the time-consuming data detection after data migration in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a data migration detection method, comprising:

[0006] Obtaining a first data migration result set and a second data migration result set in a data migration process, wherein the first data migration result set includes full migration data in a first migration file that is migrated from a migration-out host to a distributed database at a first migration time node, and the second data migration result set includes incremental migration data in a second migration file that is migrated from the migration-out host to the distributed database at a second migration time node;

[0007] Acquire a third data migration result set in the data migration process, wherein the third data migration result set includes all migration data of the migrated-out host stored in the distributed database after the data migration is completed;

[0008] Combining the first data transfer result set and the second data transfer result set to obtain a detection set;

[0009] Traversing each data in the third data migration result set, and comparing each data with the detection set, to determine a data migration detection result;

[0010] According to the data migration detection result, target indication information is generated, where the target indication information is used to instruct the distributed database to delete redundant data from the migration data stored in the migrated host.

[0011] In one embodiment, the combining the first data transfer result set and the second data transfer result set to obtain a detection set includes:

[0012] Identify target data with the same key value in the first data transfer result set and the second data transfer result set;

[0013] The first data transfer result set and the second data transfer result set are merged to obtain a detection set by retaining the target data with the same key value in the second data transfer result set and deleting the target data with the same key value in the first data transfer result set.

[0014] In one of the embodiments, before obtaining the first data migration result set and the second data migration result set in the data migration process, the method further includes:

[0015] At the first migration time node, synchronizing the first full amount of production data in the migrated host to the migration environment of the migrated host;

[0016] In the migration environment, splitting and merging the first full production data into documents to generate the first migration file;

[0017] The first migration file is imported into a distributed database and the first data migration result set is determined.

[0018] In one embodiment, the method further comprises:

[0019] Determine, according to the first migration file, a first deleted file list corresponding to the first migration time node;

[0020] In the target data table of the migrated host, the data recorded in the first deleted file list is deleted.

[0021] In one embodiment, after importing the first migration file into a distributed database and determining the first data migration result set, the method further includes:

[0022] Restarting the migration environment;

[0023] At the second migration time node, synchronizing the second incremental production data in the migrated host to the migration environment of the migrated host;

[0024] In the migration environment, splitting and merging the second incremental production data according to the second migration time node to generate the second migration file;

[0025] Import the second migration file into a distributed database and determine the second data migration result set.

[0026] In one embodiment, the method further comprises:

[0027] Within a preset time period after the second migration time node, the host to be migrated is controlled to stop executing the target service corresponding to the data to be migrated.

[0028] In one embodiment, the method further comprises:

[0029] Determine, according to the second migration file, a second deleted file list corresponding to the second migration time node;

[0030] In the target data table of the migrated host, the data recorded in the second deleted file list is deleted.

[0031] In a second aspect, the present application provides a data migration detection device, comprising:

[0032] an acquisition module, configured to acquire a first data migration result set and a second data migration result set during a data migration process, wherein the first data migration result set includes full migration data in a first migration file migrated from a migration-out host to a distributed database at a first migration time node, and the second data migration result set includes incremental migration data in a second migration file migrated from the migration-out host to the distributed database at a second migration time node; and acquire a third data migration result set during a data migration process, wherein the third data migration result set includes all migration data of the migration-out host stored in the distributed database after completing the data migration;

[0033] a merging module, configured to merge the first data transfer result set and the second data transfer result set to obtain a detection set;

[0034] A detection module, used for traversing each data in the third data migration result set, comparing each data with the detection set, and determining a data migration detection result;

[0035] The processing module is used to generate target indication information according to the data migration detection result, wherein the target indication information is used to instruct the distributed database to delete redundant data in the migration data stored in the migrated host.

[0036] In one of the embodiments, the merging module is also used to identify target data with the same key value in the first data transfer result set and the second data transfer result set; by retaining the target data with the same key value in the second data transfer result set and deleting the target data with the same key value in the first data transfer result set, the first data transfer result set and the second data transfer result set are merged to obtain a detection set.

[0037] In one of the embodiments, the processing module is also used to synchronize the first full production data in the migrated host to the migration environment of the migrated host at the first migration time node; in the migration environment, perform document splitting and merging on the first full production data to generate the first migration file; import the first migration file into a distributed database and determine the first data migration result set.

[0038] In one of the embodiments, the processing module is further used to determine a first deletion file list corresponding to the first migration time node based on the first migration file; and delete the data recorded in the first deletion file list in the target data table of the migrated host.

[0039] In one of the embodiments, the processing module is also used to restart the migration environment; at the second migration time node, synchronize the second incremental production data in the migrated host to the migration environment of the migrated host; in the migration environment, perform document splitting and merging on the second incremental production data according to the second migration time node to generate the second migration file; import the second migration file into the distributed database and determine the second data migration result set.

[0040] In one of the embodiments, the processing module is further used to control the migrated host to stop executing the target service corresponding to the data to be migrated within a preset time period after the second migration time node.

[0041] In one of the embodiments, the processing module is further used to determine a second deleted file list corresponding to the second migration time node based on the second migration file; and delete the data recorded in the second deleted file list in the target data table of the migrated host.

[0042] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the data migration detection method of the first aspect when executing the computer program.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data migration detection method of the first aspect described above.

[0044] In a fifth aspect, the present application further provides a computer program product, including a computer program, which, when executed by a processor, implements the data migration detection method of the first aspect.

[0045] The above-mentioned data migration detection method, device, equipment, storage medium and program product first obtain the first data migration result set and the second data migration result set in the data migration process, the first data migration result set includes the full migration data in the first migration file migrated from the migrating host to the distributed database at the first migration time node, and the second data migration result set includes the incremental migration data in the second migration file migrated from the migrating host to the distributed database at the second migration time node. Secondly, obtain the third data migration result set in the data migration process, the third data migration result set includes all the migration data of the migrating host stored in the distributed database after the data migration is completed. Thirdly, merge the first data transfer result set and the second data transfer result set to obtain the detection set, and traverse each data in the third data migration result set, and compare them with the detection set respectively to determine the data migration detection result. Finally, based on the data migration detection result, generate target indication information, the target indication information is used to instruct the distributed database to delete redundant data in the migration data stored in the migrating host. When performing data migration, the full migration data is first migrated once at the first migration time node, and then the incremental migration data is migrated once at the second migration time node. Therefore, after the data migration is completed, the first data migration result set corresponding to the first migration time node and the second data migration result set corresponding to the second migration time node are merged into a detection set, and then compared with the third data migration result set corresponding to all the migrated data after the data migration is completed, so as to complete the data migration detection. Since only the difference between the third data migration result set and the detection set needs to be compared, and there is no need to detect each migrated data one by one, the time spent on data detection after data migration is greatly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0047] FIG1 is an application environment diagram of a data migration detection method provided by an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a data migration detection method provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of a data migration principle provided in an embodiment of the present application;

[0050] Figure 4 A flowchart of another data migration detection method provided in an embodiment of the present application;

[0051] Figure 5 A flowchart of another data migration detection method provided in an embodiment of the present application;

[0052] Figure 6 A structural block diagram of a data migration detection device provided in an embodiment of the present application;

[0053] FIG. 7 is a diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, groups, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0055] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations. The acquisition, storage, use, and processing of data in the technical solution of this application are in compliance with relevant provisions of national laws and regulations.

[0056] The data migration detection method provided in the embodiment of the present application can be applied in the application environment shown in Figure 1. Among them, the terminal 101 communicates with the migration host 102 through the network, and the migration host 102 is connected to the server corresponding to the distributed database 103. The distributed database 103 can be integrated on the server or placed on the cloud or other network servers.

[0057] When data migration is required, the terminal 101 can send an indication message to the outgoing host 102, and the outgoing host 102 migrates the data to the distributed database 103 in batches. The outgoing host 102 can first obtain the first data migration result set and the second data migration result set in the data migration process, the first data migration result set includes the full migration data in the first migration file migrated from the outgoing host 102 to the distributed database 103 at the first migration time node, and the second data migration result set includes the incremental migration data in the second migration file migrated from the outgoing host 102 to the distributed database 103 at the second migration time node. Secondly, the outgoing host 102 obtains the third data migration result set in the data migration process, the third data migration result set includes all the migration data of the outgoing host stored in the distributed database 103 after the data migration is completed. Thirdly, the outgoing host 102 merges the first data transfer result set and the second data transfer result set to obtain a detection set, and traverses each data in the third data migration result set, and compares them with the detection set respectively to determine the data migration detection result. Finally, the host 102 to be migrated out generates target indication information according to the data migration detection result, and the target indication information is used to instruct the distributed database 103 to delete redundant data from the migration data stored in the host to be migrated out.

[0058] The terminal 101 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The migration host 102 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0059] In an exemplary embodiment, Figure 2 As shown, a data migration detection method is provided, and the method is applied to the migration-out host in FIG. 1 as an example for explanation. The data migration detection method includes S201-S205:

[0060] S201. Obtain a first data migration result set and a second data migration result set during a data migration process.

[0061] In the present application, when data migration is performed, a first data migration result set and a second data migration result set in the data migration process may be obtained, thereby detecting the data migration.

[0062] Among them, the first data migration result set includes the full migration data in the first migration file migrated from the migrated host to the distributed database at the first migration time node, and the second data migration result set includes the incremental migration data in the second migration file migrated from the migrated host to the distributed database at the second migration time node.

[0063] It should be understood that in the present application, two data migrations can be performed at the first migration time node and the second migration time node respectively, so as to migrate the business data in the migrated host to the distributed database. Among them, the first migration time node is before the second migration time node. At the first migration time node, the migrated host can migrate all the business data before the first migration time node to the distributed database without shutting down. At the second migration time node, the migrated host can migrate the incremental migration data between the first migration time node and the second migration time node to the distributed database while shutting down.

[0064] The embodiments of the present application do not limit how to obtain the first data migration result set and the second data migration result set during the data migration process. In some embodiments, after all the business data before the first migration time node is migrated to the distributed database, the first data migration result set can be obtained by the host that is migrated out. After the incremental migration data between the first migration time node and the second migration time node is migrated to the distributed database, the host that is migrated out can obtain the second data migration result set.

[0065] It should be understood that the embodiments of the present application do not limit how to migrate all the business data before the first migration time node to the distributed database. In some embodiments, the migration host can first synchronize the first full production data in the migration host to the migration environment of the migration host at the first migration time node. Subsequently, the migration host can split and merge the first full production data in the migration environment to generate a first migration file. Finally, the migration host can import the first migration file into the distributed database and determine the first data migration result set.

[0066] The first full production data may be the full business data migrated out of the host before the first migration time node.

[0067] For example, Figure 3 A schematic diagram of a data migration principle provided in an embodiment of the present application is shown as follows: Figure 3As shown, at the first migration time node (for example, one week ago), the production city disaster recovery mechanism can be used to obtain the first full production data in the migration environment of the migrated host. Subsequently, the first migration file can be generated by splitting and merging documents in the migration environment. The amount of data generated by the first migration file of the migrated device can be 1.6T, which can be completed within 1 day without shutting down the migrated device. After the first migration file is generated, the first migration file can be transcoded to make it suitable for the distributed database, and the first migration file can be downloaded and imported into the distributed database after verification, and the distributed database completes the data import within one day.

[0068] In some embodiments, the migration device may also determine a first deletion file list corresponding to the first migration time node according to the first migration file. Subsequently, the data recorded in the first deletion file list is deleted from the target data table of the migration host.

[0069] The first deleted file list may be a list of migrated data corresponding to the first migration file, and the target data table may be a data table of the business corresponding to the migrated data in the migration-out host.

[0070] It should be understood that the embodiments of the present application do not limit how to migrate the incremental migration data between the first migration time node and the second migration time node to the distributed database. In some embodiments, the migrating host may first restart the migration environment. Secondly, the migrating host may synchronize the second incremental production data in the migrating host to the migration environment of the migrating host at the second migration time node. Thirdly, the migrating host may perform document splitting and merging of the second incremental production data according to the second migration time node in the migration environment to generate a second migration file. Finally, the migrating host may import the second migration file into the distributed database and determine the second data migration result set.

[0071] Continue to refer Figure 3, at the second migration time node (for example, one week after the first migration time node), at 3 o'clock on the same day, the version can be temporarily controlled to take effect, and the execution of business services can be refused. Secondly, from 3 o'clock to 5:30 on the same day, the migration environment is restarted (2.5 hours), and the second incremental production data is obtained in the migration environment of the migrated host by using the disaster recovery mechanism in the production city. The second incremental production data can be the incremental business data between the first migration time node and the second migration time node. Thirdly, from 5:30 to 6:30 on the same day, the second migration file can be generated by document splitting and merging in the migration environment. After the second migration file is generated, the second migration file can be transcoded to adapt it to the distributed database. The generation and transcoding of the second migration file by the migrated device can be completed within 1 hour. From 6:30 to 7:30 on the same day, the second migration file is downloaded and imported into the distributed database, and the distributed database completes the data import within 1 hour. Finally, from 7:30 to 9:30 on the same day, the data verification is completed, and the business system is restored after 9:30.

[0072] In some embodiments, the migrating device may also determine a second deleted file list corresponding to the second migration time node based on the second migration file, and then the migrating device deletes the data recorded in the second deleted file list in the target data table of the migrating host.

[0073] The second deleted file list may be a list of migrated data corresponding to the second migration file.

[0074] It should be noted that the migrating host can delete the data recorded in the first deletion file list and the second deletion file list respectively, or it can merge the first deletion file list and the second deletion file list into a total deletion file list and then delete the data recorded in the total deletion file list. The embodiments of the present application do not limit this.

[0075] In some embodiments, within a preset time period after the second migration time node, the host to be migrated is controlled to stop executing the target service corresponding to the data to be migrated.

[0076] Among them, the above-mentioned preset time period can be specifically set according to the situation of the poem. For example, it can be set to 10.5 hours.

[0077] In this application, by performing multiple data migrations, the dependence of data migration time on downtime can be greatly reduced.

[0078] S202: Obtain a third data migration result set in the data migration process.

[0079] In this step, after the migration-out device obtains the first data migration result set and the second data migration result set in the data migration process, the third data migration result set in the data migration process may be obtained.

[0080] The third data migration result set includes all migration data of the migrated host stored in the distributed database after the data migration is completed. Exemplarily, the third data migration result set can be a collection of all migration data of the migrated host stored in the distributed database after the incremental migration data in the second migration file migrated from the migrated host to the distributed database at the second migration time node.

[0081] It should be understood that the embodiments of the present application provide for how to obtain the third data migration result set in the data migration process. In some embodiments, the distributed database may actively send the third data migration result set to the migrating host. In other embodiments, the migrating host may send an acquisition instruction to the distributed database to obtain the third data migration result set.

[0082] S203: Merge the first data transfer result set and the second data transfer result set to obtain a detection set.

[0083] In this step, after the migration-out device obtains the third data migration result set in the data migration process, the first data migration result set and the second data migration result set may be combined to obtain a detection set.

[0084] It should be understood that the embodiments of the present application do not limit how to merge the first data transfer result set and the second data transfer result set to obtain the detection set. In some embodiments, the migration device may first identify the target data with the same key value in the first data transfer result set and the second data transfer result set. Subsequently, the migration device retains the target data with the same key value in the second data transfer result set and deletes the target data with the same key value in the first data transfer result set, thereby merging the first data transfer result set and the second data transfer result set to obtain the detection set.

[0085] Exemplarily, the first data transfer result set obtained by the migration device is A, and the second data transfer result set is B. Then the migration device can identify the target data x with the same key value in A and B. Since the target data x appears in both A and B, but the value of the target data x may be different, in the process of data merging, the target data x with the same key value in the second data transfer result set can be retained, the target data x with the same key value in the first data transfer result set can be deleted, and other data can be merged to generate a detection set.

[0086] It should be understood that since the migrated device is still running the target business after the first migration time node, the value of the target data may change. Therefore, in the process of data merging, the target data with the same key value in the second data transfer result set can be retained, and the target data with the same key value in the first data transfer result set can be deleted, so as to determine the accuracy of the data migration.

[0087] S204 , traverse each data in the third data migration result set, compare each data with the detection set, and determine the data migration detection result.

[0088] In this step, after merging the first data migration result set and the second data migration result set to obtain a detection set, each data in the third data migration result set can be traversed and compared with the detection set respectively to determine the data migration detection result.

[0089] The data migration detection result may include data included in the third data migration result set but not included in the detection set, or data that appears repeatedly in the third data migration result set.

[0090] Exemplarily, each data in the third data migration result set may be traversed and each data may be compared with each data in the detection set, so as to determine data included in the third data migration result set but not in the detection set, or determine data that appears repeatedly in the third data migration result set.

[0091] S205. Generate target indication information according to the data migration detection result, where the target indication information is used to instruct the distributed database to delete redundant data from the migration data stored in the migrated host.

[0092] In this step, after determining the data migration detection result, the migration-out device may generate target indication information according to the data migration detection result, and the target indication information is used to instruct the distributed database to delete redundant data in the migration data stored in the migration-out host.

[0093] The redundant data may be data in the migration detection result, that is, data included in the third data migration result set but not included in the detection set, or data that appears repeatedly in the third data migration result set.

[0094] In some embodiments, the target indication information may be sent by the migration device to the distributed database, and the distributed storage repository deletes redundant data from the migration data stored in the migration host.

[0095] The data migration detection method provided in the embodiment of the present application first obtains the first data migration result set and the second data migration result set in the data migration process, wherein the first data migration result set includes the full migration data in the first migration file migrated from the migrating host to the distributed database at the first migration time node, and the second data migration result set includes the incremental migration data in the second migration file migrated from the migrating host to the distributed database at the second migration time node. Secondly, obtain the third data migration result set in the data migration process, wherein the third data migration result set includes all migration data of the migrating host stored in the distributed database after the data migration is completed. Again, merge the first data transfer result set and the second data transfer result set to obtain a detection set, and traverse each data in the third data migration result set, and compare them with the detection set respectively to determine the data migration detection result. Finally, based on the data migration detection result, generate target indication information, and the target indication information is used to instruct the distributed database to delete redundant data in the migration data stored in the migrating host. When performing data migration, the full migration data is first migrated once at the first migration time node, and then the incremental migration data is migrated once at the second migration time node. Therefore, after the data migration is completed, the first data migration result set corresponding to the first migration time node and the second data migration result set corresponding to the second migration time node are merged into a detection set, and then compared with the third data migration result set corresponding to all the migrated data after the data migration is completed, so as to complete the data migration detection. Since only the difference between the third data migration result set and the detection set needs to be compared, and there is no need to detect each migrated data one by one, the time spent on data detection after data migration is greatly reduced.

[0096] The following describes how to merge the first data transfer result set and the second data transfer result set to obtain the detection set. Figure 4 A flow chart of another data migration detection method provided in an embodiment of the present application, the data migration detection method comprises S301-S306:

[0097] S301. Obtain a first data migration result set and a second data migration result set during a data migration process.

[0098] Among them, the first data migration result set includes the full migration data in the first migration file migrated from the migrated host to the distributed database at the first migration time node, and the second data migration result set includes the incremental migration data in the second migration file migrated from the migrated host to the distributed database at the second migration time node.

[0099] S302: Obtain a third data migration result set in the data migration process.

[0100] The third data migration result set includes all migration data that is stored in the distributed database and migrated out of the host after the data migration is completed.

[0101] S303: Identify target data with the same key value in the first data transfer result set and the second data transfer result set.

[0102] S304: merging the first data transfer result set and the second data transfer result set to obtain a detection set by retaining the target data with the same key value in the second data transfer result set and deleting the target data with the same key value in the first data transfer result set.

[0103] It should be understood that since the migrated device is still running the target business after the first migration time node, the value of the target data may change. Therefore, in the process of data merging, the target data with the same key value in the second data transfer result set can be retained, and the target data with the same key value in the first data transfer result set can be deleted, so as to determine the accuracy of the data migration.

[0104] S305 , traverse each data in the third data migration result set, compare each data with the detection set, and determine the data migration detection result.

[0105] S306: Generate target indication information according to the data migration detection result, where the target indication information is used to instruct the distributed database to delete redundant data from the migration data stored in the migrated host.

[0106] In this application, data migration can be completed in batches. The last data migration can reduce the amount of migrated data as much as possible, and the requirements of completeness, accuracy, and consistency of data between tables of the migrated data can be met by using the incremental verification solution.

[0107] The following describes data migration detection in conjunction with the data migration process. Figure 5 A flowchart of another data migration detection method provided in an embodiment of the present application is provided. The data migration detection method includes S401-S416:

[0108] S401. At a first migration time node, synchronize the first full amount of production data in the migrated host to the migration environment of the migrated host.

[0109] S402: In the migration environment, split and merge the first full production data into documents to generate a first migration file.

[0110] S403: Import the first migration file into the distributed database and determine a first data migration result set.

[0111] S404: Determine a first deleted file list corresponding to the first migration time node according to the first migration file.

[0112] S405. Delete the data recorded in the first deletion file list from the target data table of the host being migrated.

[0113] S406. Restart the migration environment.

[0114] S407 . At a second migration time node, synchronize the second incremental production data in the migrated host to the migration environment of the migrated host.

[0115] S408. In the migration environment, split and merge the second incremental production data according to the second migration time node to generate a second migration file.

[0116] S409: Import the second migration file into the distributed database and determine a second data migration result set.

[0117] S410. Determine a second deleted file list corresponding to a second migration time node according to the second migration file.

[0118] S411. In the target data table of the host to be migrated, delete the data recorded in the second deletion file list.

[0119] S412: Obtain a first data migration result set and a second data migration result set during the data migration process.

[0120] Among them, the first data migration result set includes the full migration data in the first migration file migrated from the migrated host to the distributed database at the first migration time node, and the second data migration result set includes the incremental migration data in the second migration file migrated from the migrated host to the distributed database at the second migration time node.

[0121] S413: Acquire a third data migration result set in the data migration process, where the third data migration result set includes all migration data that is stored in the distributed database and migrated out of the host after the data migration is completed.

[0122] S414: Merge the first data transfer result set and the second data transfer result set to obtain a detection set.

[0123] S415 , traverse each data in the third data migration result set, compare each data with the detection set, and determine the data migration detection result.

[0124] S416. Generate target indication information according to the data migration detection result, where the target indication information is used to instruct the distributed database to delete redundant data from the migration data stored in the migrated host.

[0125] The data migration detection method provided in the embodiment of the present application, when performing data migration, first migrates the full migration data once at the first migration time node, and then migrates the incremental migration data once at the second migration time node. Therefore, after the data migration is completed, the first data migration result set corresponding to the first migration time node and the second data migration result set corresponding to the second migration time node are merged into a detection set, and then compared with the third data migration result set corresponding to all the migrated data after the data migration is completed, so as to complete the data migration detection. Since only the difference between the third data migration result set and the detection set needs to be compared, and there is no need to detect each migrated data one by one, the time spent on data detection after data migration is greatly reduced.

[0126] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0127] Based on the same inventive concept, the embodiment of the present application also provides a data migration detection device for implementing the data migration detection method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more data migration detection device embodiments provided below can refer to the limitations on the data migration detection method above, and will not be repeated here.

[0128] In an exemplary embodiment, Figure 6 As shown, a data migration detection device 500 is provided, comprising: an acquisition module 501, a merging module 502, a detection module 503 and a processing module 504, wherein:

[0129] An acquisition module 501 is used to acquire a first data migration result set and a second data migration result set in a data migration process, wherein the first data migration result set includes the full migration data in a first migration file that is migrated from a migrating host to a distributed database at a first migration time node, and the second data migration result set includes the incremental migration data in a second migration file that is migrated from a migrating host to a distributed database at a second migration time node; and acquire a third data migration result set in the data migration process, wherein the third data migration result set includes all migration data of the migrating host that is stored in the distributed database after the data migration is completed.

[0130] The merging module 502 is used to merge the first data transfer result set and the second data transfer result set to obtain a detection set.

[0131] The detection module 503 is used to traverse each data in the third data migration result set, compare each data with the detection set, and determine the data migration detection result.

[0132] The processing module 504 is used to generate target indication information according to the data migration detection result, where the target indication information is used to instruct the distributed database to delete redundant data from the migration data stored in the migrated host.

[0133] In one embodiment, the merging module 502 is also used to identify target data with the same key value in the first data transfer result set and the second data transfer result set; by retaining the target data with the same key value in the second data transfer result set and deleting the target data with the same key value in the first data transfer result set, the first data transfer result set and the second data transfer result set are merged to obtain a detection set.

[0134] In one of the embodiments, the processing module 504 is also used to synchronize the first full production data in the migrated host to the migration environment of the migrated host at the first migration time node; in the migration environment, perform document splitting and merging on the first full production data to generate a first migration file; import the first migration file into the distributed database and determine the first data migration result set.

[0135] In one of the embodiments, the processing module 504 is further used to determine a first deleted file list corresponding to the first migration time node based on the first migration file; and delete the data recorded in the first deleted file list in the target data table of the migrated host.

[0136] In one of the embodiments, the processing module 504 is also used to restart the migration environment; at a second migration time node, synchronize the second incremental production data in the migrated host to the migration environment of the migrated host; in the migration environment, split and merge the second incremental production data according to the second migration time node to generate a second migration file; import the second migration file into the distributed database and determine a second data migration result set.

[0137] In one embodiment, the processing module 504 is further configured to control the host to be migrated to stop executing the target service corresponding to the data to be migrated within a preset time period after the second migration time node.

[0138] In one of the embodiments, the processing module is further used to determine a second deleted file list corresponding to the second migration time node based on the second migration file; and delete the data recorded in the second deleted file list in the target data table of the migrated host.

[0139] Each module in the above data migration detection device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0140] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG7. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data migration detection method is implemented.

[0141] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0142] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned data migration detection method when executing the computer program.

[0143] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned data migration detection method is implemented.

[0144] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned data migration detection method when executed by a processor.

[0145] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0146] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0147] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data migration detection method, characterized in that: The method comprises: Obtaining a first data migration result set and a second data migration result set in a data migration process, wherein the first data migration result set includes full migration data in a first migration file that is migrated from a migration-out host to a distributed database at a first migration time node, and the second data migration result set includes incremental migration data in a second migration file that is migrated from the migration-out host to the distributed database at a second migration time node; Acquire a third data migration result set in the data migration process, wherein the third data migration result set includes all migration data of the migrated-out host stored in the distributed database after the data migration is completed; Combining the first data transfer result set and the second data transfer result set to obtain a detection set; Traversing each data in the third data migration result set, and comparing each data with the detection set, to determine a data migration detection result; According to the data migration detection result, target indication information is generated, where the target indication information is used to instruct the distributed database to delete redundant data from the migration data stored in the migrated host.

2. The method according to claim 1, characterized in that The step of merging the first data transfer result set and the second data transfer result set to obtain a detection set includes: Identify target data with the same key value in the first data transfer result set and the second data transfer result set; The first data transfer result set and the second data transfer result set are merged to obtain a detection set by retaining the target data with the same key value in the second data transfer result set and deleting the target data with the same key value in the first data transfer result set.

3. The method according to claim 1, characterized in that Before obtaining the first data migration result set and the second data migration result set in the data migration process, the method further includes: At the first migration time node, synchronizing the first full amount of production data in the migrated host to the migration environment of the migrated host; In the migration environment, splitting and merging the first full production data into documents to generate the first migration file; The first migration file is imported into a distributed database and the first data migration result set is determined.

4. The method according to claim 3, characterized in that The method further comprises: Determine, according to the first migration file, a first deleted file list corresponding to the first migration time node; In the target data table of the migrated host, the data recorded in the first deleted file list is deleted.

5. The method according to claim 3 or 4, characterized in that: After importing the first migration file into the distributed database and determining the first data migration result set, the method further includes: Restarting the migration environment; At the second migration time node, synchronizing the second incremental production data in the migrated host to the migration environment of the migrated host; In the migration environment, splitting and merging the second incremental production data according to the second migration time node to generate the second migration file; Import the second migration file into a distributed database and determine the second data migration result set.

6. The method according to claim 5, characterized in that The method further comprises: Within a preset time period after the second migration time node, the host to be migrated is controlled to stop executing the target service corresponding to the data to be migrated.

7. The method according to claim 5, characterized in that The method further comprises: Determine, according to the second migration file, a second deleted file list corresponding to the second migration time node; In the target data table of the migrated host, the data recorded in the second deleted file list is deleted.

8. A data migration detection device, characterized in that: The device comprises: an acquisition module, configured to acquire a first data migration result set and a second data migration result set during a data migration process, wherein the first data migration result set includes full migration data in a first migration file migrated from a migration-out host to a distributed database at a first migration time node, and the second data migration result set includes incremental migration data in a second migration file migrated from the migration-out host to the distributed database at a second migration time node; and acquire a third data migration result set during a data migration process, wherein the third data migration result set includes all migration data of the migration-out host stored in the distributed database after completing the data migration; a merging module, configured to merge the first data transfer result set and the second data transfer result set to obtain a detection set; A detection module, used for traversing each data in the third data migration result set, comparing each data with the detection set, and determining a data migration detection result; The processing module is used to generate target indication information according to the data migration detection result, wherein the target indication information is used to instruct the distributed database to delete redundant data in the migration data stored in the migrated host.

9. The device according to claim 8, characterized in that The merging module is also used to identify target data with the same key value in the first data transfer result set and the second data transfer result set; by retaining the target data with the same key value in the second data transfer result set and deleting the target data with the same key value in the first data transfer result set, the first data transfer result set and the second data transfer result set are merged to obtain a detection set.

10. The device according to claim 8, characterized in that The processing module is also used to synchronize the first full production data in the migrated host to the migration environment of the migrated host at the first migration time node; in the migration environment, perform document splitting and merging on the first full production data to generate the first migration file; import the first migration file into a distributed database and determine the first data migration result set.

11. The device according to claim 10, characterized in that The processing module is further used to determine a first deletion file list corresponding to the first migration time node based on the first migration file; and to delete the data recorded in the first deletion file list in the target data table of the migrated host.

12. The device according to claim 10 or 11, characterized in that The processing module is further configured to restart the migration environment; and synchronize the second incremental production data in the migrated host to the migration environment of the migrated host at the second migration time node; In the migration environment, splitting and merging the second incremental production data according to the second migration time node to generate the second migration file; Import the second migration file into a distributed database and determine the second data migration result set.

13. The device according to claim 12, characterized in that The processing module is further configured to control the migrated host to stop executing the target service corresponding to the data to be migrated within a preset time period after the second migration time node.

14. The device according to claim 12, characterized in that The processing module is further used to determine a second deleted file list corresponding to the second migration time node based on the second migration file; and to delete the data recorded in the second deleted file list in the target data table of the migrated host.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

17. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images