Data consistency verification method and device

Abstract

The embodiment of the invention discloses a data consistency verification method and device, and belongs to the technical field of artificial intelligence and big data. Comprising the steps that migration data are obtained from a source end database, the migration data are migrated to a target end database through a data moving operation, and the data moving operation comprises stock data and incremental data in the migration data; corresponding stock data and / or incremental data are / is obtained from the source end data stream and the target end data stream in batches and verified, a first verification result is obtained, and a difference table is determined according to the first verification result; based on the difference data in the difference table, incremental data corresponding to the difference data in the source end data stream are obtained, the incremental data corresponding to the difference data in the source end database and the target end database are inquired and verified, a second verification result is obtained, and the difference table is updated according to the second verification result; and performing consistency verification on the migration data of the source end database and the target end database according to the difference table. The data migration method and device can improve the consistency of data migration.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and big data technology, and in particular to a method and device for verifying data consistency. Background Art

[0002] With the rapid development of business and the widespread application of databases, traditional centralized databases are facing single-machine performance pressure. In related technologies, business data redistribution on newly added database nodes usually requires a lot of downtime, additional system hardware resources and a lot of manual operations. During the data migration process, data migration is prone to errors, resulting in low consistency of migrated data. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a data consistency verification method and device to solve the problem of low consistency of migration data.

[0004] To solve the above technical problems, the embodiments of the present application are implemented as follows: In a first aspect, an embodiment of the present application provides a method for verifying data consistency, comprising obtaining migration data from a source database, migrating the migration data to a target database through a data movement operation, wherein the data movement operation includes the stock data and incremental data in the migration data; respectively obtaining the corresponding stock data and / or incremental data in batches from the source data stream and the target data stream and verifying them to obtain a first verification result, determining a difference table based on the first verification result, the difference table being used to record the difference data in the first verification result about the stock data and / or incremental data corresponding to the source data stream and the target data stream; based on the difference data in the difference table, obtaining the incremental data corresponding to the difference data in the source data stream, querying and verifying the incremental data corresponding to the difference data in the source database and the target database to obtain a second verification result, and updating the difference table based on the second verification result; and performing consistency verification on the migration data of the source database and the target database according to the difference table.

[0005] In a second aspect, an embodiment of the present application provides a data consistency verification device, comprising: an acquisition module, used to acquire migration data from a source-end database, and migrate the migration data to a target-end database through a data movement operation, wherein the data movement operation includes the stock data and incremental data in the migration data; a first verification module, used to batch acquire the corresponding stock data and / or incremental data from the source-end data stream and the target-end data stream respectively and perform verification to obtain a first verification result, and determine a difference table according to the first verification result, wherein the difference table is used to record the difference data in the first verification result about the stock data and / or incremental data corresponding to the source-end data stream and the target-end data stream; a second verification module, used to acquire the incremental data corresponding to the difference data in the source-end data stream based on the difference data in the difference table, query and verify the incremental data corresponding to the difference data in the source-end database and the target-end database, obtain a second verification result, and update the difference table according to the second verification result; a determination module, used to perform consistency verification on the migration data of the source-end database and the target-end database according to the difference table.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory electrically connected to the processor, the memory storing a computer program, and the processor being used to call and execute the computer program from the memory to implement the above-mentioned data consistency verification method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program can be executed by a processor to implement the above-mentioned data consistency verification method.

[0008] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the above-mentioned data consistency verification method.

[0009] By adopting the technical solution of the embodiment of the present application, migration data is obtained from the source database, and the migration data is migrated to the target database through a data movement operation, wherein the data movement operation includes the stock data and incremental data in the migration data; the corresponding stock data and / or incremental data are respectively obtained in batches from the source data stream and the target data stream and verified to obtain a first verification result, without the need for additional data storage, and the data can be compared while reading; a difference table is determined according to the first verification result, and the difference table is used to record the difference data in the first verification result about the stock data and / or incremental data corresponding to the source data stream and the target data stream; based on the difference data in the difference table, the incremental data corresponding to the difference data in the source data stream is obtained, the incremental data corresponding to the difference data in the source database and the target database are queried and verified to obtain a second verification result, and the difference table is updated according to the second verification result; according to the difference table, the migration data of the source database and the target database are consistency checked. It can be seen that the difference data in the difference table is determined by the first verification result, and the incremental data corresponding to the difference data in the source database and the target database are verified based on the difference data. This improves the integrity of the migrated data in the source database and the target database without stopping the data for processing and without occupying hardware resources, so as to solve the problem of low consistency of the migrated data. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate one or more embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0011] Figure 1 is a schematic flow chart of a data consistency verification method according to an embodiment of the present application; Figure 2 is a schematic diagram of a process of converting batch-read data according to an embodiment of the present application; Figure 3 is a schematic diagram of a process of merging batch-read data according to an embodiment of the present application; Figure 4 is a schematic diagram of a process for real-time consistency verification of data according to an embodiment of the present application; Figure 5 is a schematic diagram of a data flow in a business system according to an embodiment of the present application; Figure 6 is a schematic flow chart of a scenario of a data consistency verification method according to an embodiment of the present application; Figure 7 is a schematic block diagram of a data consistency verification device according to an embodiment of the present application; Figure 8 It is a hardware structure diagram of a data consistency verification device according to an embodiment of the present application. DETAILED DESCRIPTION

[0012] The embodiments of the present application provide a data consistency verification method and device to solve the problem of low accuracy of data consistency verification.

[0013] In order to enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0014] The data consistency verification method provided in the embodiment of the present application can be executed by an electronic device, or by software installed in the electronic device. Specifically, the electronic device can be a terminal device or a server device. Among them, the terminal device can include a smart phone, a laptop computer, a smart wearable device, a vehicle terminal, etc., and the server device can include an independent physical server, a server cluster composed of multiple servers, or a cloud server capable of cloud computing.

[0015] In the following, in conjunction with the accompanying drawings, a data consistency verification method provided by an embodiment of the present application is described in detail through specific embodiments and application scenarios.

[0016] Figure 1 A schematic flow chart of a data consistency verification method provided by an embodiment of the present invention is shown, and the method comprises the following steps: S102, obtaining migration data from the source database, and migrating the migration data to the target database through a data migration operation.

[0017] The data movement operation includes the migration of existing data and incremental data.

[0018] The source database refers to the original source or original storage location of the migrated data. In contrast to the target database, the source database provides the original data to be processed or analyzed, while the target database stores the processed, processed, and analyzed data.

[0019] Stock data refers to the total amount of data accumulated over a certain period of time and stored in job task items in a database or data system.

[0020] Incremental data refers to data that has changed or been updated over a period of time, where the changes can be new data records, modified data records, or deleted data records. It is usually used in data warehouses, databases, and data processing systems to more efficiently store and process growing data.

[0021] The data movement job is used to migrate the existing data and incremental data in the migration data to the target database.

[0022] S104, respectively obtain the corresponding stock data and / or incremental data in batches from the source data stream and the target data stream and perform verification to obtain a first verification result, and determine a difference table according to the first verification result.

[0023] The difference table is used to record the difference data of the stock data and / or incremental data corresponding to the source data stream and the target data stream in the first verification result.

[0024] Source data flow refers to the data flow from the data source, i.e. the source database, during the data transmission process. Data flow refers to the transmission path and process of data in a system or network, which can be real-time and continuous, or discrete and intermittent.

[0025] Target end data flow refers to the data flow path and process from the sender to the target end during data transmission.

[0026] The differential data may be the differential data between the stock data of the source data stream and the stock data of the target data stream corresponding to the stock data, or the differential data between the incremental data of the source data stream and the incremental data of the target data stream corresponding to the incremental data.

[0027] It should be noted that the corresponding stock data and / or incremental data are obtained in batches from the source data stream and the target data stream and verified, including: first verifying the stock data, and verifying the incremental data in real time, the stock data is historical data, and the incremental data is real-time data, such as incremental data can be data modified from the stock data, or data newly added to the stock data, etc. In other words, the corresponding stock data can be obtained in batches from the source data stream and the target data stream, and the stock data can be verified, and then the corresponding incremental data can be obtained in batches from the source data stream and the target data stream, and the incremental data can be verified, and at this time, there is no need to verify the stock data; the corresponding stock data can also be obtained in batches from the source data stream and the target data stream, and the stock data can be verified, and at the same time, the corresponding incremental data can be obtained in batches from the source data stream and the target data stream, and the incremental data can be verified in real time.

[0028] The first verification result includes verifying the corresponding stock data obtained in batches from the source data stream and the target data stream, and obtaining the verification result, such as data that the corresponding stock data in the source data stream and the target data stream are the same, or difference data that the corresponding stock data in the source data stream and the target data stream are different. And / or verifying the corresponding incremental data obtained in batches from the source data stream and the target data stream, and obtaining the verification result, such as data that the corresponding incremental data in the source data stream and the target data stream are the same, or difference data that the corresponding incremental data in the source data stream and the target data stream are different.

[0029] The difference data in the first verification result about the stock data and / or incremental data corresponding to the source data stream and the target data stream is recorded in a difference table without the need for additional data storage.

[0030] S106, based on the difference data in the difference table, obtain the incremental data corresponding to the difference data in the source data stream, query and verify the incremental data corresponding to the difference data in the source database and the target database, obtain a second verification result, and update the difference table according to the second verification result.

[0031] According to the difference data in the difference table, the incremental data corresponding to the difference data is obtained from the source data stream, and according to the difference data in the difference table, the incremental data corresponding to the difference data in the source database and the target database are queried, and based on the incremental data corresponding to the difference data in the source data stream and the incremental data corresponding to the difference data in the source database and the target database, the data in the source database and the target database are verified to obtain a second verification result.

[0032] It should be noted that when the difference data is the difference data between the stock data corresponding to the source data stream and the target data stream, the data in the source database and the target database corresponding to the difference data can be directly obtained for verification and comparison, and the difference table is updated after the second verification result is obtained. Among them, if there is a problem in the process of migrating data, the corresponding stock data in the target database is modified directly according to the record of the stock data in the source database; if it is due to modifying the stock data in the source database corresponding to the difference data, then it is necessary to obtain the incremental data corresponding to the difference data in the source data stream, and then verify the migration data in the source database and the target database again to obtain the second verification result.

[0033] When the difference data is the difference data between the incremental data corresponding to the source data stream and the target data stream, the migration data in the source database and the target database are verified directly based on the incremental data corresponding to the difference data obtained from the source data stream, the incremental data corresponding to the difference data in the source database, and the incremental data corresponding to the difference data in the target database to obtain a second verification result.

[0034] The second verification result includes the same data or different data of the incremental data of the source database and the target database corresponding to the difference data. The incremental data of the source database and the target database include the incremental data that generates the difference data in the difference table and the migration data at the corresponding position in the source database and the target database.

[0035] According to the second verification result, the difference data in the difference table is updated. For example, when the incremental data corresponding to the difference data in the source database and the target database are consistent, the difference data in the difference table is deleted.

[0036] S108: Perform consistency check on the migration data of the source database and the target database according to the difference table.

[0037] Since the difference table records the difference data of the stock data and / or incremental data corresponding to the source data stream and the target data stream in the first verification result, after determining the incremental data in the source data stream and the incremental data corresponding to the source database and the target database based on the difference data, a consistency check is performed. If the incremental data corresponding to the source database and the target database are the same, the difference data is deleted. It can be seen that when all the migration data of the source database and the target database are consistent, the difference table is empty.

[0038] By adopting the technical solution of the embodiment of the present application, migration data is obtained from the source database, and the migration data is migrated to the target database through a data movement operation, wherein the data movement operation includes the stock data and incremental data in the migration data; the corresponding stock data and / or incremental data are respectively obtained in batches from the source data stream and the target data stream and verified to obtain a first verification result, without the need for additional data storage, and the data can be compared while reading; a difference table is determined according to the first verification result, and the difference table is used to record the difference data in the first verification result about the stock data and / or incremental data corresponding to the source data stream and the target data stream; based on the difference data in the difference table, the incremental data corresponding to the difference data in the source data stream is obtained, the incremental data corresponding to the difference data in the source database and the target database are queried and verified to obtain a second verification result, and the difference table is updated according to the second verification result; according to the difference table, the migration data of the source database and the target database are consistency checked. It can be seen that the difference data in the difference table is determined by the first verification result, and the incremental data corresponding to the difference data in the source database and the target database are verified based on the difference data. This improves the integrity of the migrated data in the source database and the target database without stopping the data for processing and without occupying hardware resources, so as to solve the problem of low consistency of the migrated data.

[0039] In one embodiment, the migration data is migrated to the target database (ie, S102) through a data migration operation, and the following steps A1-A3 may be performed: Step A1: submit the data movement job to the job scheduling subsystem according to the preset data redistribution rule.

[0040] The data redistribution rule refers to redistributing the data movement operations while keeping the data from being lost and backed up unchanged.

[0041] Step A2: split the data movement job into multiple job task items according to the job scheduling subsystem, and each job task item includes corresponding stock data and incremental data.

[0042] The job scheduling subsystem can split the data movement job according to the submitted job configuration based on the data redistribution rule to obtain multiple job task items. Specifically, the data movement job is split into multiple job task items, and the tasks of each job task item can be executed simultaneously in a distributed manner, thereby improving the speed of data redistribution in the data movement job.

[0043] Step A3: Migrate the migration data to the target database according to each job task item.

[0044] In this embodiment, the acquired data movement job is split and processed through the job scheduling subsystem to obtain multiple job task items, which can be executed in a distributed manner. That is, the corresponding existing data and incremental data in the multiple job task items can be migrated to the target database at the same time, which can improve the speed of migrating the migrated data and is conducive to processing massive data.

[0045] In one embodiment, each job task item is migrated to the target database (i.e., step A3), and the following steps B1-B3 may be performed: Step B1, storing the incremental data corresponding to each job task item into the incremental migration subsystem, and storing the stock data corresponding to each job task item into the stock migration subsystem, to obtain the stock data and incremental data of the data movement job.

[0046] Optionally, the incremental migration subsystem also has a breakpoint resume subsystem, which can continue to store the incremental data corresponding to the job task item in the incremental migration subsystem when the incremental data corresponding to the job task item is interrupted and then restarted, without having to start the transmission from the beginning, and can resume the transmission from the breakpoint. Similarly, the stock migration subsystem also has a breakpoint resume subsystem, which can continue to store the stock data corresponding to the job task item in the stock migration subsystem when the stock data corresponding to the job task item is interrupted and then restarted.

[0047] Step B2, obtaining the incremental progress through the breakpoint resume subsystem in the incremental migration subsystem, and obtaining the stock progress through the breakpoint resume subsystem in the stock migration subsystem.

[0048] According to step B1, the incremental migration subsystem is used to obtain the incremental data in each job task item, and the stock migration subsystem is used to obtain the stock data in each job task item. According to step B2, the incremental progress in each job task item is obtained through the breakpoint resume subsystem that also exists in the incremental migration subsystem, and the stock progress in each job task item is obtained through the breakpoint resume subsystem that also exists in the incremental migration subsystem. Since both the incremental migration subsystem and the stock incremental migration subsystem have breakpoint resume subsystems, when the incremental progress or stock progress is interrupted and restarted, the incremental progress or stock progress can continue to be obtained.

[0049] Step B3, storing the stock data and incremental data into the target database, and writing the incremental progress and stock progress into the registration center of the corresponding business system through the progress storage subsystem.

[0050] The incremental data obtained by the incremental migration subsystem and the stock data obtained by the stock migration subsystem are written into the target database through the efficient writing subsystem; the incremental progress and the stock progress are written into the registration center of the corresponding business system through the progress storage subsystem, wherein the incremental progress and the stock progress can be persisted in time, that is, when the data movement job is restarted, the data movement job can be continued from the previous progress.

[0051] The progress storage subsystem can obtain the incremental progress and the stock progress and record them in the registration center. Through the registration center, the incremental progress of the incremental data and the stock progress of the stock data can be obtained in a timely manner.

[0052] In this embodiment, the incremental migration subsystem and the stock migration subsystem can store the acquired incremental data and stock data into the target database through the efficient writing subsystem; they can also write the acquired incremental progress and stock progress into the registration center of the corresponding business system through the progress storage subsystem, and realize breakpoint resumption of the acquisition progress and / or stock progress by recording the progress, that is, the breakpoint position of the acquisition of incremental data and / or stock data can be determined through the recorded progress, and breakpoint resumption can be realized, thereby improving the speed of the redistribution data movement operation and ensuring the continuity of the business's external services.

[0053] In one embodiment, corresponding stock data and / or incremental data are obtained in batches from the source data stream and the target data stream respectively and verified to obtain a first verification result, and a difference table is determined according to the first verification result (i.e., S104), and the following steps C1-C2 may be performed: Step C1, batch acquiring first stock data in the source data stream and second stock data in the target data stream, and / or batch acquiring first incremental data in the source data stream and second incremental data in the target data stream.

[0054] It should be understood that the first stock data obtained in the source data stream and the second stock data obtained in the target data stream correspond to each other, that is, the first stock data sent by the source data stream is the second stock data obtained in the target data stream. Similarly, the acquisition method of incremental data is the same as that of stock data. Generally, historical data, i.e., stock data, is acquired first, and incremental data is acquired at the same time. Alternatively, stock data can be acquired first and then incremental data can be acquired. There is no specific limitation.

[0055] Step C2, verifying the first stock data and the second stock data, and / or verifying the first incremental data and the second incremental data to obtain a first verification result. When the first verification result includes difference data, the difference data is recorded in a difference table.

[0056] The verification process includes: comparing each record in the first stock data and the second stock data, and / or comparing each record in the first incremental data and the second incremental data.

[0057] The first stock data and the second stock data are verified through the stock verification subsystem, and the verification progress is stored in the registration center through the progress storage subsystem; the first incremental data and the second incremental data are verified through the incremental verification subsystem, and the verification progress is stored in the registration center through the progress storage subsystem. The verification progress includes the verification progress of each record in the stock data and / or incremental data.

[0058] By comparing each record of the first stock data and the second stock data one by one, and / or comparing each record of the first incremental data and the second incremental data one by one, a first verification result can be determined, and the first verification result includes records in which the first stock data and the second stock data are identical and records in which there are difference data, and / or records in which the first incremental data and the second incremental data are identical and records in which there are difference data.

[0059] The records of difference data between the first stock data and the second stock data, and / or the records of difference data between the first incremental data and the second incremental data are stored in a difference table.

[0060] Based on the verification progress, since the stock verification subsystem also has a breakpoint-resume subsystem, when the verification process of the first stock data and the second stock data is interrupted and then restarted, the verification process can continue from the interruption point, without the need to verify the first stock data and the second stock data from the beginning, and the verification can be continued from the breakpoint. Similarly, the incremental verification subsystem also includes a breakpoint-resume subsystem, and the first incremental data and the second incremental data can also achieve breakpoint-resume comparison.

[0061] In this embodiment, by batch acquiring the first stock data in the source data stream and the second stock data in the target data stream, and / or batch acquiring the first incremental data in the source data stream and the second incremental data in the target data stream, and performing verification processing, a first verification result is obtained, and the difference data included in the first verification result is recorded in a difference table. No additional data storage is required, which can improve the resource space of the business system, and by storing the difference data in the first verification result in the difference table, the difference data can be centrally processed to improve processing efficiency.

[0062] In one embodiment, the incremental data corresponding to the difference data in the source database and the target database is queried and verified to obtain a second verification result, and the difference table is updated according to the second verification result (ie, S106), and the following steps D1-D2 may be performed: Step D1, according to the incremental data corresponding to the differential data in the source data stream, query the incremental data corresponding to the differential data in the source database and the target database, and verify the incremental data corresponding to the differential data in the source database and the target database to obtain a second verification result.

[0063] Since the difference data is recorded in the difference table, there is no need to compare each stock data or incremental data. The corresponding data of the difference data in the source database and the target database can be verified directly based on the records of the difference data in the difference table.

[0064] Specifically, when the difference data in the difference table is the difference data between the stock data of the source data stream and the stock data of the target data stream, it may be a problem that occurs during the transmission of the stock data in the source database when migrating to the target database. Based on the difference data, the stock data corresponding to the difference data in the source database and the target database are determined for comparison to determine whether it is a problem that occurs during the migration process, rather than new incremental data generated by modifying the stock data in the source database. If the difference data is generated by modifying the stock data in the source database, the incremental data corresponding to the difference data in the source data stream is determined, and the incremental data corresponding to the difference data in the source database and the target database are compared. When the incremental data corresponding to the difference data in the source database and the target database are consistent, and are also consistent with the incremental data corresponding to the difference data in the source data stream, the second verification result is that the incremental data corresponding to the difference data in the source database and the target database are consistent, and the relevant records of the difference data in the difference table are deleted.

[0065] When the difference data is the difference data between the incremental data of the source data stream and the incremental data of the target data stream, since there are differences in the incremental data, the incremental data in the source database may be modified or the incremental data is newly added data. Therefore, based on the difference data, the incremental data corresponding to the difference data in the source data stream is determined, and the incremental data corresponding to the difference data in the source database and the target database are compared. When the incremental data corresponding to the difference data in the source database and the target database are consistent, and are also consistent with the incremental data corresponding to the difference data in the source data stream, the second verification result is that the incremental data corresponding to the difference data in the source database and the target database are consistent, and the relevant records of the difference data in the difference table are deleted.

[0066] Since the existing data is historical data, after the existing data is verified, the incremental data is read and verified at the same time, and there is no need to repeat the verification of the existing data.

[0067] Step D2: When the second verification result shows that the incremental data corresponding to the difference data in the source database and the target database are consistent, the corresponding difference data in the difference table is deleted.

[0068] The difference data existing in the stock data and / or incremental data recorded in the difference table are verified one by one to obtain a second verification result. When the corresponding incremental data in the source database and the target database corresponding to each difference data are consistent, the difference record is deleted. When the stock data and / or incremental data in the source database and the target database are completely consistent, the record in the difference table is empty. If the second verification result is that the corresponding incremental data in the source database and the target database corresponding to the difference data are inconsistent, the record of the difference data continues to be saved, or a new record of the difference data is added.

[0069] Preferably, in the process of performing consistency check on the data in the source database and the target database, the difference table is scanned according to a preset time. When there is no difference between the migration data in the source database and the target database corresponding to the difference data in the difference table, the difference data recorded in the difference table is deleted; when after multiple rounds of scanning the difference table, if it is detected that the difference table is empty, it is determined that the migration data in the source database and the target database are consistent.

[0070] In this embodiment, the incremental data in the source data stream is determined based on the difference records in the difference table, and the stock data and / or incremental data corresponding to the source database and the target database are verified in turn based on each difference data recorded in the difference table. When the stock data and / or incremental data corresponding to the source database and the target database are consistent, the difference data is deleted. When the stock data and / or incremental data corresponding to the source database and the target database are completely consistent, the record in the difference table is empty. The efficiency of verifying the source database and the target database can be improved.

[0071] In one embodiment, first stock data in a source data stream and second stock data in a target data stream are obtained in batches, and / or first incremental data in a source data stream and second incremental data in a target data stream are obtained in batches (i.e., step C1), and specifically, the following steps E1-E2 may be performed: Step E1, converting the data in the first stock data and the second stock data into multiple records, and / or converting the data in the first incremental data and the second incremental data into multiple records.

[0072] The source data stream includes a source stock data stream and / or a source incremental data stream, including first stock data and second stock data, and / or first incremental data and second incremental data. Among a large amount of data, the first stock data and second stock data, and / or the first incremental data and second incremental data are relatively large. The first stock data and second stock data, and / or the first incremental data and second incremental data are subjected to data conversion and shaping processing to obtain corresponding multiple records.

[0073] Specifically, a schematic diagram of a conversion process for batch read data is shown as follows: Figure 2 As shown, the streaming sorting and aggregation is performed from each shard, such as the shard is a data file obtained from each database, that is, a database file (datebase, DB), and a database file includes stock data or incremental data. For example, different shards are DB1, DB2, and DB3, and the obtained migration data is shaped and converted into multiple records, such as the data in DB1 is recorded as record K1, record K2...record K000 in sequence; the data in DB2 is recorded as record 1001, record 1002...record 2000 in sequence; the data in DB3 is recorded as record 1, record 2...record 1000 in sequence. When checking the consistency of data, each piece of data is recorded for checking the migration data, and the stock data and / or incremental data are read in batches, so that the reading efficiency of the stock data and / or incremental data can be improved.

[0074] Step E2, merging the multiple data to obtain multiple records corresponding to the first stock data and multiple records corresponding to the second stock data, and / or obtaining multiple records corresponding to the first incremental data and multiple records corresponding to the second incremental data.

[0075] The merging process includes: merging records with the same value of the first field of the primary key together for processing.

[0076] Due to the excessive amount of existing data and / or incremental data, multiple records are generated. For multi-field primary key tables, data segmentation query can only use the first field. To prevent data from being filtered out by other fields, some records with the same first field value will be generated. During paging query, only part of them may be queried. The condition of the next segmentation query will skip the remaining records, resulting in this part of the data not being verified. Therefore, compatibility is achieved through merging processing.

[0077] Specifically, a schematic diagram of a process of merging batch read data is shown as follows: Figure 3As shown, the multiple data recorded in step E1 are merged, for example, each corresponding DB record is merged, and the records with the same first field are merged, wherein each merged record has a unique feature point to avoid missing data during the verification process. For example, the existing data stream is merged to obtain the corresponding merged record K000-id: K000; record K-id: K; record 1-id: 1, record 2-id: 2, record 3-id: 3.

[0078] In this embodiment, multiple records are obtained by converting the existing data and / or incremental data obtained in batches, and the multiple records are merged, that is, the records with the same value of the first field of the primary key are merged together for processing, which can avoid the problem of missing data in the process of verifying the migrated data.

[0079] In one embodiment, after consistency check is performed on the migration data of the source database and the target database according to the difference table (ie, S108), the following step F may be further performed: Step F: after determining that the migration data of the source database and the target database are consistent, the source database in the business system is switched to the target database by modifying the metadata.

[0080] According to the data after confirming that the migration data of the source database and the target database are consistent, that is, after verifying that the migration data in the source database and the target database correspond to each other; modifying the metadata can be to modify the information stored in the target database, or to modify the situation of the database used in the business system. For example, since the migration data in the source database is migrated to the target database, it is necessary to switch the business using the source database to the target database, such as by modifying the metadata, changing the link of the source database specified in the business system to the link of the target database. Through distributed coordination, the metadata in the business system cluster is unified and effective, and the business system completes the data redistribution process.

[0081] In this embodiment, the metadata can be switched after the verification is passed. Once the switch is completed, the business-unaware re-sharding is completed, reducing the impact of the data movement job migration on the processing of other related business work.

[0082] In one embodiment, a flow chart of real-time consistency check of data is as follows: Figure 4As shown, the source-side data stream includes the source-side stock data stream and the source-side incremental data stream, and the target-side data stream also includes the target-side stock data stream and the target-side incremental data stream. Specifically, step 1.1, batch obtain each record of the first stock data in the source-side stock data stream, and batch obtain each record of the second stock data in the target-side data stream, and compare each record one by one; step 1.2, when encountering records with different primary keys, since the data is merged, the data segments are loaded and then compared one by one; step 1.3, the end position of the comparison verification can be recorded through the breakpoint-resume subsystem, and the comparison will continue from the interruption position later. The comparison result is used as the first verification result, and the difference data in the first verification result is recorded as the newly added difference data in the difference table.

[0083] Among them, after the existing data is verified, the incremental data is verified in real time, that is, the incremental data is read and verified in real time, and the verification result is used as the first verification result, and the difference data in the first verification result is also recorded in the difference table as new difference data.

[0084] According to the difference table, query the existing difference records, step 2, determine the difference data according to the difference records, obtain the incremental data corresponding to the difference data in the source data stream according to the difference data, and the records of the incremental data corresponding to the difference data in the source database and the target database, and process the migration data in the source database and the target database corresponding to the difference data. If the processing is not successful, add new difference data to the difference table. If the processing is successful, delete the difference record of the difference data in the difference table. Step 2 also includes, when there is a problem when the stock data in the source database is migrated to the target database, resulting in the existence of difference records of difference data in the stock data, directly process the difference records corresponding to the source database and the target database, such as adjusting the corresponding stock data in the target database according to the stock data in the source database. If the processing is successful, delete the difference records related to the difference data.

[0085] The updated difference table is tested for multiple rounds. If the difference table records are empty after multiple rounds of testing, it means that the migration data in the source database and the target database are completely consistent, and the consistency check of the migration data in the source database and the target database passes.

[0086] In one embodiment, a schematic diagram of data flow in a business system is shown as follows: Figure 5As shown, the migration data is obtained and a data movement job is generated, and the data movement job is re-sharded. According to the configuration of the mobile data service, such as the job scheduling subsystem, the data movement job is divided into multiple job task items. Each job task item may include existing data and incremental data, such as job task item 1, job task item 2, and job task item n. Based on the incremental task configuration and the existing task configuration, the incremental data and incremental progress are obtained through the incremental migration subsystem and the breakpoint resume subsystem, the existing data and existing progress are obtained through the stock migration subsystem and the breakpoint resume subsystem, and the existing data and incremental data are written to the target database through the efficient writing subsystem. The stock progress and incremental progress are written to the registration center through the progress storage subsystem. The registration center feeds back the incremental progress and stock progress to the incremental migration subsystem and the stock migration subsystem in real time.

[0087] After the data movement job is re-sharded, the existing data and incremental data are checked for consistency in real time. The existing data is checked for consistency through the existing verification subsystem and the breakpoint-resume subsystem, and then the incremental data is checked for consistency through the incremental verification subsystem and the breakpoint-resume subsystem. The progress of the existing verification and the progress of the incremental verification are stored in the progress storage subsystem and written to the registration center. The registration center can also feed back the incremental verification progress and the progress of the existing verification to the incremental verification subsystem and the existing verification subsystem in real time. After the real-time consistency check of the existing data and the incremental data in the source database and the target database is completed, the subsystem is switched by modifying the metadata, and the switched subsystem is written to the registration center to switch the source database used in the business system to the target database.

[0088] Figure 6 is a schematic flow chart of a scenario of a data consistency verification method according to another embodiment of the present application, such as Figure 6 As shown, the method comprises the following steps: S601, obtain migration data from the source database, generate a data movement job, and submit the data movement job to the job scheduling subsystem according to a preset data redistribution rule.

[0089] S602: Split the data movement job into multiple job task items according to the job scheduling subsystem.

[0090] Among them, each job task item includes corresponding stock data and incremental data.

[0091] S603, storing the incremental data corresponding to each job task item into the incremental migration subsystem, and storing the stock data corresponding to each job task item into the stock migration subsystem, and storing into the target database through the efficient writing subsystem.

[0092] S604, obtaining the incremental progress according to the breakpoint resume subsystem in the incremental migration subsystem, and obtaining the stock progress according to the breakpoint resume subsystem in the stock migration subsystem, and writing the incremental progress and the stock progress into the registration center of the business system through the progress storage subsystem.

[0093] S605, batch-acquire the stock data in the source data stream and the stock data in the target data stream by means of merging processing, and / or batch-acquire the incremental data in the source data stream and the incremental data in the target data stream.

[0094] S606, verifying the corresponding stock data and / or incremental data obtained in batches from the source data stream and the target data stream to obtain a first verification result.

[0095] Since the stock data is historical data, after verifying the stock data, you can directly verify the incremental data without repeating the verification of the stock data.

[0096] S607, determining a difference table according to the first verification result, where the difference table is used to record difference data in the first verification result about the stock data and / or incremental data corresponding to the source data stream and the target data stream.

[0097] S608, based on the difference data in the difference table, obtain the incremental data corresponding to the difference data in the source data stream, and query the incremental data corresponding to the difference data in the source database and the target database.

[0098] S609, verify the incremental data corresponding to the difference data in the source database and the target database through the stock verification subsystem and / or the incremental verification subsystem to obtain a second verification result, and update the difference table according to the second verification result.

[0099] Among them, the stock verification subsystem and the incremental verification subsystem include a breakpoint-resume comparison subsystem, and the breakpoint-resume comparison subsystem is used to record the comparison progress related to the difference data in the source database and the target database.

[0100] S610: Perform consistency check on the migration data of the source database and the target database according to the difference table.

[0101] S611, when the difference data in the difference table is empty, that is, the migration data of the source database and the target database are consistent, the source database in the business system is switched to the target database by modifying the metadata.

[0102] The specific process from S601 to S611 has been described in detail in the above embodiment and will not be repeated here.

[0103] By adopting the technical solution of the embodiment of the present application, migration data is obtained from the source database, and the migration data is migrated to the target database through a data movement operation, wherein the data movement operation includes the stock data and incremental data in the migration data; the corresponding stock data and / or incremental data are respectively obtained in batches from the source data stream and the target data stream and verified to obtain a first verification result, without the need for additional data storage, and the data can be compared while reading; a difference table is determined according to the first verification result, and the difference table is used to record the difference data in the first verification result about the stock data and / or incremental data corresponding to the source data stream and the target data stream; based on the difference data in the difference table, the incremental data corresponding to the difference data in the source data stream is obtained, the incremental data corresponding to the difference data in the source database and the target database are queried and verified to obtain a second verification result, and the difference table is updated according to the second verification result; according to the difference table, the migration data of the source database and the target database are consistency checked. It can be seen that the difference data in the difference table is determined by the first verification result, and the incremental data corresponding to the difference data in the source database and the target database are verified based on the difference data. This improves the integrity of the migrated data in the source database and the target database without stopping the data for processing and without occupying hardware resources, so as to solve the problem of low consistency of the migrated data.

[0104] In summary, specific embodiments of the present subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recorded in the claims can be performed in a different order and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing can be advantageous.

[0105] The above is a data consistency verification method provided in an embodiment of the present application. Based on the same idea, an embodiment of the present application also provides a data consistency verification device.

[0106] Figure 7 is a schematic diagram of the structure of a data consistency verification device according to an embodiment of the present invention. Figure 7 As shown, the data consistency verification device includes: an acquisition module 71, a first verification module 72, a second verification module 73, and a determination module 74: The acquisition module 71 is used to acquire the migration data from the source database and migrate the migration data to the target database through a data migration operation. The data migration operation includes the stock data and the incremental data in the migration data. A first verification module 72 is used to obtain corresponding stock data and / or incremental data in batches from the source data stream and the target data stream respectively and perform verification to obtain a first verification result, and determine a difference table according to the first verification result, the difference table being used to record difference data in the first verification result about the stock data and / or incremental data corresponding to the source data stream and the target data stream; A second verification module 73 is used to obtain incremental data corresponding to the difference data in the source data stream based on the difference data in the difference table, query and verify the incremental data corresponding to the difference data in the source database and the target database, obtain a second verification result, and update the difference table according to the second verification result; The determination module 74 is used to perform consistency check on the migration data of the source database and the target database according to the difference table.

[0107] In one embodiment, the acquisition module 71 is specifically used to submit the data movement job to the job scheduling subsystem according to the preset data redistribution rules; split the data movement job into multiple job task items according to the job scheduling subsystem, each job task item includes corresponding existing data and incremental data; migrate the migration data to the target database according to each of the job task items.

[0108] In one embodiment, the acquisition module 71 includes migrating each job task item to the target database, specifically storing the incremental data corresponding to each job task item into the incremental migration subsystem, and storing the stock data corresponding to each job task item into the stock migration subsystem, to obtain the stock data and incremental data of the data movement job; obtaining the incremental progress through the breakpoint resume subsystem in the incremental migration subsystem, and obtaining the stock progress through the breakpoint resume subsystem in the stock migration subsystem; storing the stock data and incremental data in the target database, and writing the incremental progress and stock progress to the registration center of the corresponding business system through the progress storage subsystem.

[0109] In one embodiment, the first verification module 72 includes: An acquisition unit, used for batch acquiring first stock data in a source data stream and second stock data in a target data stream, and / or batch acquiring first incremental data in a source data stream and second incremental data in a target data stream; The first verification unit is used to verify the first stock data and the second stock data, and / or verify the first incremental data and the second incremental data to obtain a first verification result. When the first verification result includes difference data, the difference data is recorded in a difference table, wherein the verification processing includes: comparing each record in the first stock data and the second stock data, and / or comparing each record in the first incremental data and the second incremental data.

[0110] In one embodiment, the second verification module 73 specifically includes: A query unit, configured to query the incremental data corresponding to the difference data in the source-end database and the target-end database according to the incremental data corresponding to the difference data in the source-end data stream, and verify the incremental data corresponding to the difference data in the source-end database and the target-end database to obtain a second verification result; The second verification unit is used to delete the corresponding difference data in the difference table when the second verification result shows that the incremental data corresponding to the difference data in the source database and the target database are consistent.

[0111] In one embodiment, the acquisition unit is specifically used to convert the data in the first stock data and the second stock data into multiple records, and / or convert the data in the first incremental data and the second incremental data into multiple records; merge the multiple data to obtain multiple records corresponding to the first stock data and multiple records corresponding to the second stock data, and / or obtain multiple records corresponding to the first incremental data and multiple records corresponding to the second incremental data, wherein the merging process includes: merging records with the same value of the first field of the primary key together for processing.

[0112] In one embodiment, the device further includes a switching module for switching the source database in the business system to the target database by modifying metadata based on data after determining that the migration data of the source database and the target database are consistent.

[0113] By adopting the technical solution of the embodiment of the present application, migration data is obtained from the source database, and the migration data is migrated to the target database through a data movement operation, wherein the data movement operation includes the stock data and incremental data in the migration data; the corresponding stock data and / or incremental data are respectively obtained in batches from the source data stream and the target data stream and verified to obtain a first verification result, without the need for additional data storage, and the data can be compared while reading; a difference table is determined according to the first verification result, and the difference table is used to record the difference data in the first verification result about the stock data and / or incremental data corresponding to the source data stream and the target data stream; based on the difference data in the difference table, the incremental data corresponding to the difference data in the source data stream is obtained, the incremental data corresponding to the difference data in the source database and the target database are queried and verified to obtain a second verification result, and the difference table is updated according to the second verification result; according to the difference table, the migration data of the source database and the target database are consistency checked. It can be seen that the difference data in the difference table is determined by the first verification result, and the incremental data corresponding to the difference data in the source database and the target database are verified based on the difference data. This improves the integrity of the migrated data in the source database and the target database without stopping the data for processing and without occupying hardware resources, so as to solve the problem of low consistency of the migrated data.

[0114] Those skilled in the art should understand that Figure 7 The data consistency verification device in can be used to implement the data consistency verification method described above, and the detailed description thereof should be similar to the description of the method part above. To avoid tediousness, it will not be repeated here.

[0115] Based on the same technical concept, an embodiment of the present application further provides an electronic device, which is used to perform the above-mentioned data consistency verification method. Figure 8 A schematic diagram of the structure of an electronic device for implementing various embodiments of the present application. The electronic device may have relatively large differences due to different configurations or performances, and may include a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call a computer program stored in the memory 830 and executable on the processor 810 to perform the following steps: Obtain the migration data from the source database and migrate it to the target database through data migration operations. The data migration operations include the existing data and incremental data in the migration data. respectively acquiring corresponding stock data and / or incremental data in batches from the source-end data stream and the target-end data stream and performing verification to obtain a first verification result, and determining a difference table according to the first verification result, the difference table being used to record difference data in the first verification result about the stock data and / or incremental data corresponding to the source-end data stream and the target-end data stream; Based on the difference data in the difference table, the incremental data corresponding to the difference data in the source data stream is obtained, the incremental data corresponding to the difference data in the source database and the target database are queried and verified to obtain a second verification result, and the difference table is updated according to the second verification result; Based on the difference table, perform consistency check on the migrated data of the source database and the target database.

[0116] By adopting the technical solution of the embodiment of the present application, migration data is obtained from the source database, and the migration data is migrated to the target database through a data movement operation, wherein the data movement operation includes the stock data and incremental data in the migration data; the corresponding stock data and / or incremental data are respectively obtained in batches from the source data stream and the target data stream and verified to obtain a first verification result, without the need for additional data storage, and the data can be compared while reading; a difference table is determined according to the first verification result, and the difference table is used to record the difference data in the first verification result about the stock data and / or incremental data corresponding to the source data stream and the target data stream; based on the difference data in the difference table, the incremental data corresponding to the difference data in the source data stream is obtained, the incremental data corresponding to the difference data in the source database and the target database are queried and verified to obtain a second verification result, and the difference table is updated according to the second verification result; according to the difference table, the migration data of the source database and the target database are consistency checked. It can be seen that the difference data in the difference table is determined by the first verification result, and the incremental data corresponding to the difference data in the source database and the target database are verified based on the difference data. This improves the integrity of the migrated data in the source database and the target database without stopping the data for processing and without occupying hardware resources, so as to solve the problem of low consistency of the migrated data.

[0117] The specific execution steps can refer to the various steps of the above-mentioned data consistency verification method embodiment, and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0118] It should be noted that the electronic devices in the embodiments of the present application include: servers, terminals, or other devices except terminals.

[0119] The above electronic device structure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. For example, the input unit may include a graphics processing unit (GPU) and a microphone, and the display unit may be configured with a display panel in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit includes a touch panel and at least one of other input devices. The touch panel is also called a touch screen. Other input devices may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0120] The memory can be used to store software programs and various data. The memory may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory may include a volatile memory or a non-volatile memory, or the memory may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DRRAM).

[0121] The processor may include one or more processing units; optionally, the processor integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to the operating system, user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor.

[0122] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned data consistency verification method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0123] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0124] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned data consistency verification method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0125] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0126] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the processor is used to run the program or instructions to implement the various processes of the above-mentioned product recommendation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0127] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0128] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0129] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for verifying data consistency, characterized in that: The method comprises: Acquire migration data from a source database, and migrate the migration data to a target database through a data migration operation, wherein the data migration operation includes stock data and incremental data in the migration data; respectively acquiring the corresponding stock data and / or the incremental data in batches from the source-end data stream and the target-end data stream and performing verification to obtain a first verification result, and determining a difference table according to the first verification result, wherein the difference table is used to record difference data in the first verification result about the stock data and / or the incremental data corresponding to the source-end data stream and the target-end data stream; Based on the difference data in the difference table, obtaining incremental data corresponding to the difference data in the source data stream, querying and verifying the incremental data corresponding to the difference data in the source database and the target database to obtain a second verification result, and updating the difference table according to the second verification result; According to the difference table, a consistency check is performed on the migration data of the source database and the target database.

2. The method according to claim 1, characterized in that The step of migrating the migration data to the target database through the data migration operation includes: Submitting the data movement job to the job scheduling subsystem according to a preset data redistribution rule; Splitting the data movement job into a plurality of job task items according to the job scheduling subsystem, each of the job task items including corresponding stock data and incremental data; The migration data is migrated to the target database according to each of the job task items.

3. The method according to claim 2, characterized in that Migrating each of the job task items to the target database includes: The incremental data corresponding to each of the job task items is stored in the incremental migration subsystem, and the stock data corresponding to each of the job task items is stored in the stock migration subsystem, so as to obtain the stock data and the incremental data of the data migration job; Acquiring incremental progress through the breakpoint-resume subsystem in the incremental migration subsystem, and acquiring stock progress through the breakpoint-resume subsystem in the stock migration subsystem; The stock data and the incremental data are stored in the target-side database, and the incremental progress and the stock progress are written into the registration center of the corresponding business system through the progress storage subsystem.

4. The method according to claim 1, characterized in that: The step of respectively acquiring corresponding stock data and / or incremental data in batches from the source data stream and the target data stream and performing verification to obtain a first verification result, and determining a difference table according to the first verification result includes: Batch acquiring the first stock data in the source end data stream and the second stock data in the target end data stream, and / or batch acquiring the first incremental data in the source end data stream and the second incremental data in the target end data stream; The first stock data and the second stock data are verified and / or the first incremental data and the second incremental data are verified to obtain the first verification result; when the first verification result includes the difference data, the difference data is recorded in the difference table, wherein the verification process includes: comparing each record in the first stock data and the second stock data, and / or comparing each record in the first incremental data and the second incremental data.

5. The method according to claim 1, characterized in that The querying and verifying the incremental data corresponding to the difference data in the source database and the target database to obtain a second verification result, and updating the difference table according to the second verification result, includes: According to the incremental data corresponding to the difference data in the source data stream, query the incremental data corresponding to the difference data in the source database and the target database, and verify the incremental data corresponding to the difference data in the source database and the target database to obtain the second verification result; When the second verification result is that the incremental data corresponding to the difference data in the source database and the target database are consistent, the corresponding difference data in the difference table is deleted.

6. The method according to claim 4, characterized in that The batch acquiring of the first stock data in the source end data stream and the second stock data in the target end data stream, and / or the batch acquiring of the first incremental data in the source end data stream and the second incremental data in the target end data stream, comprises: Converting the data in the first stock data and the second stock data into a plurality of stock data records, and / or converting the data in the first incremental data and the second incremental data into a plurality of incremental data records; The multiple records of the existing data and / or the multiple records of the incremental data are merged to obtain multiple records corresponding to the first existing data and multiple records corresponding to the second existing data, and / or multiple records corresponding to the first incremental data and multiple records corresponding to the second incremental data, wherein the merging process includes: merging records with the same value of the first field of the primary key together for processing.

7. The method according to claim 1, characterized in that After performing consistency check on the migration data of the source database and the target database according to the difference table, the method further includes: According to the data after determining that the migration data of the source database and the target database are consistent, the source database in the business system is switched to the target database by modifying the metadata.

8. A data consistency verification device, characterized in that: The device comprises: An acquisition module is used to acquire migration data from a source database and migrate the migration data to a target database through a data migration operation, wherein the data migration operation includes stock data and incremental data in the migration data; A first verification module, used to batch-acquire the stock data and / or the incremental data from the source-end data stream and the target-end data stream respectively and perform verification to obtain a first verification result, and determine a difference table according to the first verification result, wherein the difference table is used to record difference data in the first verification result about the stock data and / or the incremental data corresponding to the source-end data stream and the target-end data stream; A second verification module is used to obtain incremental data corresponding to the difference data in the source data stream based on the difference data in the difference table, query and verify the incremental data corresponding to the difference data in the source database and the target database to obtain a second verification result, and update the difference table according to the second verification result; A determination module is used to perform consistency check on the migration data of the source database and the target database according to the difference table.

9. An electronic device, characterized in that: It comprises a processor and a memory electrically connected to the processor, the memory stores a computer program, and the processor is used to call and execute the computer program from the memory to implement a data consistency verification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium is used to store a computer program, and the computer program can be executed by a processor to implement a data consistency verification method as described in any one of claims 1 to 7.

Images