Data checking method and device and nonvolatile storage medium

Abstract

The invention discloses a data checking method and device and a nonvolatile storage medium. The method comprises the steps of collecting first incremental data of a source database and second incremental data of a target database; data matched with the corresponding identifier information included in the incremental data is sent to the corresponding processing threads; processing the first incremental data in a first processing thread to obtain a plurality of first operation types and first information fields corresponding to the source database; processing the second incremental data in a second processing thread to obtain a plurality of second operation types and second information fields corresponding to the target database; and based on the first operation type, the first information field of the second operation type and the second information field of the second operation type, checking the first incremental data and the second incremental data to obtain a data checking result. According to the method and the device, the technical problems of low dual-system data checking efficiency and insufficient accuracy in the migration process of the distributed system at present are solved.

Description

Technical Field

[0001] This invention relates to the field of distributed system migration technology, and more specifically, to a data verification method, apparatus, and non-volatile storage medium. Background Technology

[0002] During the migration of the system technology stack from a centralized system to a distributed system, the sheer volume of data—reaching hundreds of millions of records daily and storing nearly ten billion records—and the processing time covering 24 hours a day, makes data accuracy and consistency verification paramount. To ensure the accuracy of data migration, a serial dual-write approach can be used to run both the centralized and distributed systems, with the centralized system data serving as the benchmark for verification. However, during this process, the accuracy of transactions in the distributed system cannot be manually verified one by one; therefore, building a real-time monitoring system to check data consistency is essential.

[0003] Current technologies, when dealing with data volumes of hundreds of millions, suffer from excessively long processing times and low efficiency when using centralized comparison methods. The current batch processing model struggles to meet the verification requirements of real-time data changes, exhibiting poor real-time performance. Furthermore, traditional architectures lack scalability, making it difficult to flexibly adapt to data volume fluctuations and easily becoming performance bottlenecks. In addition, in a distributed environment, current technologies offer insufficient guarantees for data consistency and have inadequate bypass system design, failing to meet the need for efficient and accurate data comparison and migration without affecting the normal operation of the main business system. This also fails to fully address the specific data consistency monitoring requirements during distributed system migration.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a data verification method, apparatus, and non-volatile storage medium to at least solve the technical problems of low efficiency and insufficient accuracy in dual-system data verification during distributed system migration.

[0006] According to one aspect of the present invention, a data verification method is provided, comprising: collecting first incremental data from a source database and second incremental data from a target database, wherein the source database is a database of a centralized system, the target database is a database of a distributed system, the first incremental data is data that changes in the source database during system migration, and the second incremental data is data that changes in the target database during system migration; sending data matching first identifier information included in the first incremental data to a first processing thread, and sending data matching second identifier information included in the second incremental data to a second processing thread; in the first processing thread, recording and parsing the first incremental data to obtain multiple first operation types corresponding to the source database and first information fields corresponding to each of the multiple first operation types; in the second processing thread, recording and parsing the second incremental data to obtain multiple second operation types corresponding to the target database and second information fields corresponding to each of the multiple second operation types; and verifying the first incremental data and the second incremental data based on the first operation type, the second operation type, the first information field, and the second information field to obtain a data verification result, wherein the data verification result includes data matching success, data matching failure, and unconfirmed.

[0007] Optionally, sending data matching the first identifier information included in the first incremental data to the first processing thread includes: constructing multiple first message partitions; delivering the first incremental data to the multiple first message partitions respectively, wherein data corresponding to the same first identifier information is delivered to the same first message partition; extracting data from the multiple first message partitions respectively to construct a first data table to be processed; calculating the first hash value of the data in the first data table to be processed according to the database name of the source database and the first identifier information; and sending the data in the first data table to be processed to the first processing thread based on the first hash value.

[0008] Optionally, sending data matching the second identifier information included in the second incremental data to the second processing thread includes: constructing multiple second message partitions; delivering the second incremental data to the multiple second message partitions respectively, wherein data corresponding to the same second identifier information is delivered to the same second message partition; extracting data from the multiple second message partitions respectively to construct a second data table to be processed; calculating the second hash value of the data in the second data table to be processed based on the database name of the target database and the second identifier information; and sending the data in the second data table to be processed to the second processing thread based on the second hash value.

[0009] Optionally, in the first processing thread, the first incremental data is recorded and parsed to obtain multiple first operation types corresponding to the source database and the first information fields corresponding to each of the multiple first operation types, including: recording and parsing the first incremental data to obtain multiple first operation types and first information fields, wherein the first operation type includes an add type, an update type, or a delete type, the first information fields corresponding to the add type and the update type are subsequent information fields, and the first information field corresponding to the delete type is a preceding information field.

[0010] Optionally, in the second processing thread, the second incremental data is recorded and parsed to obtain multiple second operation types corresponding to the target database and the second information fields corresponding to each of the multiple second operation types. This includes: after obtaining the first information field, the second incremental data is recorded and parsed to obtain multiple second operation types and second information fields. The second operation types include add type, update type, or delete type. The second information fields corresponding to add type and update type are subsequent information fields, and the second information fields corresponding to delete type are preceding information fields.

[0011] Optionally, based on the first operation type, the first information field of the second operation type, and the second information field, the first incremental data and the second incremental data are checked to obtain a data check result, including: determining the operation type that matches the multiple first operation types among the multiple second operation types, and respectively using it as the first target operation type and the second target operation type; calculating the hash value of the data corresponding to the first target operation type to obtain a third hash value; calculating the hash value of the data corresponding to the second target operation type to obtain a fourth hash value; comparing whether the third hash value and the fourth hash value match; if the third hash value and the fourth hash value do not match, determining that the data check result is a data matching failure, and comparing the first information field and the second information field to obtain a field difference result; and recording the field difference result.

[0012] Optionally, it also includes: if the data verification result shows that the data matching fails, obtaining the correspondence between the operation type and the business identification code; determining the target business identification code corresponding to the second incremental data based on the correspondence; and sending a warning message to the user corresponding to the target business identification code based on the level of the target business identification code, wherein the warning message includes a warning level.

[0013] According to another aspect of the present invention, a data verification apparatus is also provided, comprising: a collection module, configured to collect first incremental data from a source database and second incremental data from a target database, wherein the source database is a database of a centralized system, the target database is a database of a distributed system, the first incremental data is data that changes in the source database during system migration, and the second incremental data is data that changes in the target database during system migration; a sending module, configured to send data matching first identifier information included in the first incremental data to a first processing thread, and send data matching second identifier information included in the second incremental data to a second processing thread; a first parsing module, configured to record and parse the first incremental data in the first processing thread to obtain multiple first operation types corresponding to the source database and first information fields corresponding to each of the multiple first operation types; a second parsing module, configured to record and parse the second incremental data in the second processing thread to obtain multiple second operation types corresponding to the target database and second information fields corresponding to each of the multiple second operation types; and a verification module, configured to verify the first incremental data and the second incremental data based on the first operation type, the second operation type, the first information field, and the second information field to obtain a data verification result, wherein the data verification result includes data matching success, data matching failure, and unconfirmed.

[0014] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is running, it controls the device where the non-volatile storage medium is located to execute any of the above data verification methods.

[0015] According to another aspect of the present invention, a computer device is also provided, the computer device including a processor, the processor being configured to run a program, wherein the program executes any of the above-described data verification methods during runtime.

[0016] In this embodiment of the invention, a data verification method is employed. This involves collecting first incremental data from a source database and second incremental data from a target database. The source database is a centralized system database, and the target database is a distributed system database. The first incremental data represents data that changed in the source database during system migration, and the second incremental data represents data that changed in the target database during system migration. Data matching the first identifier information included in the first incremental data is sent to a first processing thread, and data matching the second identifier information included in the second incremental data is sent to a second processing thread. In the first processing thread, the first incremental data is recorded and parsed to obtain multiple first operation types corresponding to the source database, and each of the multiple first operation types... The first information field corresponds to the second incremental data. In the second processing thread, the second incremental data is recorded and parsed to obtain multiple second operation types corresponding to the target database and the second information fields corresponding to each of the multiple second operation types. Based on the first operation type, the second operation type, the first information field, and the second information field, the first incremental data and the second incremental data are checked to obtain the data check result. The data check result includes data matching success, data matching failure, and unconfirmed. This achieves the goal of improving real-time monitoring capabilities and ensuring data consistency in a distributed system migration environment, thereby improving the efficiency and accuracy of data check. This solves the technical problem of low efficiency and insufficient accuracy of dual-system data check in the current distributed system migration process. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 A hardware structure block diagram of a computer terminal for implementing a data verification method is shown.

[0019] Figure 2 This is a flowchart illustrating the data verification method provided according to an embodiment of the present invention;

[0020] Figure 3 This is an architecture diagram of a data verification system provided by an optional embodiment of the present invention;

[0021] Figure 4 This is a structural diagram of a monitoring system provided according to an optional embodiment of the present invention;

[0022] Figure 5 This is a structural block diagram of a data verification device provided according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0026] Serial dual-write: A business logic is executed sequentially in both the new and old systems after it reaches the core system.

[0027] A+B stage: A is a centralized system, B is a distributed system. A+B represents a stage where both the centralized and distributed systems execute the same business logic, but the data is based on the centralized system.

[0028] B+A stage: A is a centralized system, B is a distributed system, and B+A represents a stage where both the distributed and centralized systems execute the same business logic, but the data is based on the distributed system.

[0029] Kafka Message Queue: Kafka is a distributed messaging system developed by LinkedIn. It can be used to build real-time data pipelines and streaming applications, supports integration with various distributed systems, and enables real-time data transmission between the same or different application systems.

[0030] Incremental data refers to newly generated or updated data (not full data) in a data source, which is usually identified by timestamps, version numbers, or log tags.

[0031] According to an embodiment of the present invention, a data verification method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a data verification method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data verification method in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the data verification method of the application described above. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0035] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0036] Figure 2 This is a flowchart illustrating the data verification method provided in an embodiment of the present invention, as shown below. Figure 2 As shown, the method includes the following steps:

[0037] Step S201: Collect the first incremental data of the source database and the second incremental data of the target database. The source database is a database of a centralized system, and the target database is a database of a distributed system. The first incremental data is the data that changes in the source database during system migration, and the second incremental data is the data that changes in the target database during system migration.

[0038] In this step, the source database refers to the traditional centralized database that is currently in the system migration process and serves as the core for data processing and storage, used to store data from centralized system A. The target database refers to the distributed database used by the new system after the migration, used to store data from distributed system B.

[0039] Specifically, at each stage of the system migration, whenever a new business operation occurs or existing data is modified, the source and target databases will independently perform data write operations based on database log parsing technology. The source database can monitor its redo log using OGG (Oracle Golden Gate) technology, extracting data changes in real time and converting them into JSON format for delivery to the Kafka cluster. The target database can collect its database logs using Dbbridge technology. After data is written to the corresponding database, corresponding incremental data is generated. The first incremental data contains data that changed in the source database during the system migration, and the second incremental data contains data that changed in the target database during the system migration. This incremental data includes timestamped change records in the format {table name, primary key, operation type, field value, timestamp}, which can be identified by a unique identifier field and stored in the Kafka cluster according to the data source. An example of the message format in the Kafka cluster is as follows:

[0040] {

[0041] "table ": "user_info",

[0042] "pk": "user_id=123",

[0043] "op ": "UPDATE",

[0044] "data" : {"before":{ "user_nam":"name1"}, "after": { "user_nam":"name2"}},

[0045] "timestamp": "2025-04-20 14:30:00"

[0046] }

[0047] Step S202: Data matching the first identifier information included in the first incremental data is sent to the first processing thread, and data matching the second identifier information included in the second incremental data is sent to the second processing thread.

[0048] In this step, the first identifier refers to the unique identifier of each data record in the source database, i.e., the primary key field in the business table, such as user ID, account number, etc. This information is encapsulated in a Kafka message during the data acquisition phase and transmitted along with the incremental data. Similarly, the second identifier is the unique identifier of the data record in the target database, also based on the primary key field of the business table, ensuring the uniqueness and traceability of the data records.

[0049] Specifically, this optional embodiment constructs dual-topic business tables (Topic-A and Topic-B) in the Kafka cluster module to store incremental data from the source and target databases, respectively. Based on the decoupling of Kafka topic message consumption and processing, the incremental data in each topic is partitioned to form Kafka messages, and the JSON-formatted Kafka message bodies are converted into the respective segment values ​​of the data tables and stored in the message pending processing table. Subsequently, the data in the message pending processing table is processed by the data processing thread. By assigning specific sharding keys to each message processing instance, it can be ensured that data with the same identifier can be processed in the same instance. Within each processing instance, multi-threaded parallel processing is further employed. Thread allocation is based on the hash value of the first or second identifier information in the message to ensure that data with the same identifier is assigned to the same thread for processing. Specifically, data with the first identifier is assigned to the first processing thread, which is dedicated to processing the first incremental data; data with the second identifier is assigned to the second processing thread, which is dedicated to processing the second incremental data.

[0050] Step S203: In the first processing thread, the first incremental data is recorded and parsed to obtain multiple first operation types corresponding to the source database and the first information fields corresponding to each of the multiple first operation types.

[0051] In this step, the first operation type refers to the various operations performed on the source database by the centralized system when processing transactions. The first information field is a data item in the Kafka message body corresponding to each first operation type, recording all behavioral data for each first operation type. By recording and parsing the first incremental data, the monitoring system can identify and separate the operation type of each record, and further extract the information fields associated with that operation type. The above operation requires decoding the JSON format data in the Kafka message, converting it into a homogeneous business table structure defined in the monitoring system, saving the operation type marker, and filling the corresponding first information field with specific field values.

[0052] Step S204: In the second processing thread, the second incremental data is recorded and parsed to obtain multiple second operation types corresponding to the target database and the second information fields corresponding to each of the multiple second operation types.

[0053] In this step, the second operation type refers to the various operations performed on the target database by the distributed system when processing business. The second information field is a data item in the Kafka message body corresponding to each second operation type, recording all behavioral data for each second operation type. The recording and parsing of the second incremental data are processed using a delayed processing method based on the time records in the message record table. By recording and parsing the second incremental data, the monitoring system can identify and separate the operation type of each record, and further extract the information fields associated with that operation type. The above operations require decoding the JSON format data in the Kafka message, converting it into a homogeneous business table structure defined in the monitoring system, saving the operation type marker, and filling the corresponding second information field with specific field values.

[0054] Step S205: Based on the first operation type, the second operation type, the first information field, and the second information field, the first incremental data and the second incremental data are checked to obtain the data check result, wherein the data check result includes data matching success, data matching failure, and unconfirmed.

[0055] In this step, data verification is performed based on the consistency confirmation of operation types in the two databases and the precise comparison of related field values. Specifically, based on whether the identifiers are consistent, a query is made to see if a first operation type corresponding to each second operation type exists. If it exists, the corresponding information fields of the two operation types with the same identifier are compared and verified. To avoid redundant comparisons, a pre-calculated hash value is used for rapid pre-screening. Further, the information fields of the above data are first converted, concatenated into strings, and the string hash value is calculated. If the hash values ​​of both sides are consistent, it is marked as a successful match; otherwise, it is marked as a failed match. For the data that fails to match, a field-by-field verification is performed, and the difference value is recorded in the remarks field. Conversely, if no first operation type with the same identifier exists, it is marked as unconfirmed.

[0056] For the three data verification results mentioned above, a successful data match indicates that a completely consistent corresponding record has been found between the source database and the target database, proving the effectiveness of the data migration and serial dual-write process; a failed data match indicates that there are differences or errors, and further investigation and repair are required; while an unconfirmed status indicates that the data processing is delayed, and the system needs to wait for a period of time or take a retry mechanism in order to more comprehensively evaluate the data consistency.

[0057] Through the above steps, the goal of improving real-time monitoring capabilities and ensuring data consistency in a distributed system migration environment is achieved, thereby improving the efficiency and accuracy of data verification and solving the technical problems of low efficiency and insufficient accuracy of dual-system data verification during the current distributed system migration process.

[0058] As an optional embodiment, this can be achieved through the following steps: sending data matching the first identifier information included in the first incremental data to the first processing thread, including: constructing multiple first message partitions; delivering the first incremental data to the multiple first message partitions respectively, wherein data corresponding to the same first identifier information is delivered to the same first message partition; extracting data from the multiple first message partitions respectively to construct a first data table to be processed; calculating the first hash value of the data in the first data table to be processed according to the database name of the source database and the first identifier information; and sending the data in the first data table to be processed to the first processing thread based on the first hash value.

[0059] Optionally, constructing multiple first message partitions involves creating corresponding topics for all business tables in the source database within the Kafka cluster. The number of partitions for each topic is set based on the average daily incremental data volume and peak incremental data volume of each table, resulting in a maximum of 48 first message partitions. Next, the Kafka message producer performs a hash operation based on the first identifier of the business table. The result determines the partition to which the data belongs, and data with the same result is delivered to the same first message partition, achieving precise data delivery. Then, the monitoring system pulls data from the Kafka cluster by partition, converts JSON-formatted messages into structured records, and constructs the first table to be processed. Subsequently, 48 message processing instances are created, each assigned a shard key. Each instance reads the data to be processed from the first table according to its shard key and sorts it by timestamp. A hash operation is performed based on the database name and first identifier information of the source database to obtain a fixed-length digest representing the data characteristics, i.e., the first hash value. Finally, based on the first hash value, the monitoring system sends data with the same first hash value to the same first processing thread, ensuring that data with the same first identifier information (i.e., primary key) in the same topic can be processed in the same thread, thereby improving data processing speed and overall system throughput.

[0060] As an optional embodiment, this can be achieved through the following steps: sending data matching the second identifier information included in the second incremental data to the second processing thread, including: constructing multiple second message partitions; delivering the second incremental data to the multiple second message partitions respectively, wherein data corresponding to the same second identifier information is delivered to the same second message partition; extracting data from the multiple second message partitions respectively to construct a second data table to be processed; calculating the second hash value of the data in the second data table to be processed based on the database name of the target database and the second identifier information; and sending the data in the second data table to be processed to the second processing thread based on the second hash value.

[0061] Optionally, similar to the processing of the first incremental data described above, constructing multiple second message partitions involves creating corresponding topics for all business tables of the target database within the Kafka cluster. The number of partitions for each topic is set based on the average daily incremental data volume and peak incremental data volume of each table, thus constructing second message partitions, with a maximum of 48 partitions. Next, the Kafka message producer performs a hash operation based on the second identifier of the business table, determines the partition to which the data belongs based on the operation result, and then delivers data with the same result to the same second message partition. Then, the monitoring system pulls data from the Kafka cluster by partition, converts the JSON-formatted messages into structured records, and constructs the second data table to be processed. Subsequently, 48 message processing instances are created, each assigned a shard key to be processed. Each message processing instance reads the data to be processed from the second data table according to its own shard key and sorts it by timestamp. A hash operation is performed based on the database name and second identifier information of the target database to obtain a fixed-length digest representing the data characteristics, i.e., the second hash value. Finally, based on the second hash value, the monitoring system sends data with the same second hash value to the same second processing thread, ensuring that data with the same second identifier information (i.e., primary key) in the same topic can be processed in the same thread.

[0062] As an optional embodiment, this can be achieved through the following steps: In the first processing thread, the first incremental data is recorded and parsed to obtain multiple first operation types corresponding to the source database and the first information fields corresponding to each of the multiple first operation types, including: recording and parsing the first incremental data to obtain multiple first operation types and first information fields, wherein the first operation type includes an add type, an update type, or a delete type, the first information fields corresponding to the add type and the update type are subsequent information fields, and the first information field corresponding to the delete type is a preceding information field.

[0063] Optionally, after receiving records from the message processing table, the first processing thread transforms these records into operable database objects. This process involves decoding the JSON-formatted Kafka message and extracting the first operation type contained within it. The first operation type includes INSERT, UPDATE, or DELETE operations.

[0064] Specifically, for INSERT and UPDATE operations, the first information field is the after field. After parsing the INSERT or UPDATE operation, the processing thread needs to use the value of the after field in the Kafka message as its first information field. The after field records the content after the data change in detail. For INSERT operations, it contains the complete data value of the new record; for UPDATE operations, it contains the latest data value of the modified field. For DELETE operations, the first information field is the before field. After parsing the DELETE operation, the processing thread needs to use the value of the before field in the Kafka message as its first information field. The before field stores the state before the data change; for DELETE operations, it contains the data value deleted from the database.

[0065] As an optional embodiment, this can be achieved through the following steps: In the second processing thread, the second incremental data is recorded and parsed to obtain multiple second operation types corresponding to the target database and the second information fields corresponding to each of the multiple second operation types, including: after obtaining the first information field, the second incremental data is recorded and parsed to obtain multiple second operation types and second information fields, wherein the second operation types include add type, update type or delete type, the second information fields corresponding to add type and update type are subsequent information fields, and the second information fields corresponding to delete type are preceding information fields.

[0066] Optionally, after receiving records from the message pending processing table, the second processing thread performs delayed processing based on the preset time interval of the first information field and the time records in the message record table, transforming the records into operable database objects. This process involves decoding the JSON-formatted Kafka message and extracting the second operation type contained in the Kafka message. The second operation type includes INSERT, UPDATE, or DELETE operations.

[0067] Specifically, similar to the first information field, the second information field for INSERT and UPDATE operations is the after field. After parsing the INSERT or UPDATE operation, the processing thread needs to use the value of the after field in the Kafka message as its second information field. The after field records the content after the data change in detail. For INSERT operations, it contains the complete data value of the new record; for UPDATE operations, it contains the latest data value of the modified field. The second information field for DELETE operations is the before field. After parsing the DELETE operation, the processing thread needs to use the value of the before field in the Kafka message as its second information field. The before field stores the state before the data change; for DELETE operations, it contains the data value deleted from the database.

[0068] As an optional embodiment, this can be achieved through the following steps: Based on a first operation type, a first information field of a second operation type, and a second information field, verify the first incremental data and the second incremental data to obtain a data verification result, including: determining the operation type that matches one of the multiple first operation types among multiple second operation types, respectively as the first target operation type and the second target operation type; calculating the hash value of the data corresponding to the first target operation type to obtain a third hash value; calculating the hash value of the data corresponding to the second target operation type to obtain a fourth hash value; comparing whether the third hash value and the fourth hash value match; if the third hash value and the fourth hash value do not match, determining the data verification result as a data matching failure, and comparing the first information field and the second information field to obtain a field difference result; and recording the field difference result.

[0069] Optionally, firstly, a matching first operation type is retrieved based on the second operation type. If multiple matching operation types exist, they are used as the first and second target operation types, respectively. Secondly, to reduce data verification redundancy, hash value comparison is used to simplify the data. Data in the first target operation type is concatenated into a string, and a hash value is calculated for the string as the third hash value. Data in the second target operation type is concatenated into a string, and a hash value is calculated for the string as the fourth hash value. Subsequently, the third and fourth hash values ​​are compared to see if they match (are the same). If the third and fourth hash values ​​do not match, the data verification result is determined to be a data matching failure. Finally, for the first and second incremental data that failed to match, the first information field and the second information field are compared to obtain the field difference results, and these results are recorded in the abnormal data table.

[0070] As an optional embodiment, this can be achieved through the following steps: further including: if the data verification result is that the data matching fails, obtaining the correspondence between the operation type and the business identification code; based on the correspondence, determining the target business identification code corresponding to the second incremental data; and based on the level of the target business identification code, sending a warning message to the user corresponding to the target business identification code, wherein the warning message includes a warning level.

[0071] Optionally, all SQL statements for database operations involving transaction codes (i.e., business identifier codes) can be pre-analyzed to establish a mapping relationship between the operation types of the business tables and the transaction codes. Simultaneously, the transaction volume, transaction attributes (financial, maintenance, etc.), and developers for each transaction code can be pre-analyzed, and the importance of each transaction code can be assessed. If the data verification result shows a data matching failure, an anomaly analysis mechanism is activated. First, a pre-defined mapping table is retrieved to obtain the correspondence between operation types and business identifier codes. Second, based on the above correspondence, the target business identifier code corresponding to the operation type in the second incremental data of the anomaly is obtained. Finally, the business identifier codes are divided into three levels: "important," "moderate," and "minor," and based on the level of the target business identifier code, a warning message is sent to the user corresponding to the target business identifier code. The warning message includes the warning level, which is the level of the aforementioned business identifier code.

[0072] As an optional embodiment, Figure 3 This is an architecture diagram of a data verification system provided according to an optional embodiment of the present invention. For example... Figure 3As shown, this optional embodiment consists of System A, System B, Monitoring System Y, and a Kafka cluster. System A is a centralized system, and System B is a distributed system. System A includes an application, a source database, and a data acquisition module; System B includes an application, a target database, and a data acquisition module. The monitoring system includes modules for data reception, data processing, and result feedback. Systems A and B first capture incremental data changes in the source and target databases in real time through the data acquisition module, ensuring that every data addition, deletion, and modification is accurately recorded and converted into a standardized JSON message format. Subsequently, the message is transferred to the Kafka cluster for storage and distribution. The Kafka cluster receives and stores incremental data messages sent from System A and System B through dual topics (Topic-A and Topic-B), respectively. Kafka's distributed nature and high throughput enable it to efficiently handle large-scale message streams. Simultaneously, through an identifier delivery strategy, it ensures that incremental data messages with the same identifier are delivered to the same partition. Finally, the monitoring system Y pulls messages from the Kafka cluster through the data receiving module and places them in the message pending processing table. The data processing module then processes messages with the same identifier. Based on a multi-threaded parallel processing mechanism and a hash distribution strategy, the data processing module combines a "hash pre-screening + field fine-tuning" algorithm to achieve data processing and difference detection, effectively improving the efficiency and accuracy of data comparison. The result feedback module verifies the results and outputs them to the data table in real time, marking the data consistency status, recording difference details, and is equipped with an alarm system to ensure that abnormal data can be detected in a timely manner and notified to relevant developers for rapid response and handling.

[0073] As an optional embodiment, Figure 4 This is a structural diagram of a monitoring system provided according to an optional embodiment of the present invention. Figure 4 As shown, firstly, the consumer instance group pulls incremental data messages stored in the Kafka cluster, which includes dual-write data from both centralized and distributed systems. Then, the Kafka message bodies pulled by the consumer instances are added to the message waiting table, awaiting further processing. The table-adding operation uses a hash distribution strategy to ensure that all messages with the same identifier are mapped to the same partition, thus being handled by a unified message processing instance.

[0074] Specifically, the monitoring system comprises 48 instances, each tasked with processing messages from a specific partition. During processing, the instances first sort the messages by timestamp, then convert the JSON fields in the messages into values ​​for business tables. Finally, processing threads are allocated based on the hash value of the combination of the table name and a unique identifier, ensuring ordered processing and efficient comparison of messages with the same identifier. Messages from System A (Topic-A) are processed immediately, while messages from System B (Topic-B) are processed with a five-minute delay. Furthermore, the processing threads calculate the hash values ​​of matching operation types for different topic messages. When hash values ​​do not match, the differences are compared and recorded in an anomaly data table for subsequent anomaly analysis and location.

[0075] Finally, by analyzing the transaction logs and business tables, the monitoring system identifies the transaction codes that generated abnormal data. Based on the importance of the transaction codes, it assesses the warning level and then sends warning information to the relevant personnel for the corresponding transaction codes, ensuring a rapid response and resolution of the problem.

[0076] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0077] Through the above description of the embodiments, those skilled in the art can clearly understand that the data verification method according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0078] According to embodiments of the present invention, an apparatus for implementing the above-described data verification method is also provided. Figure 5 This is a structural block diagram of a data verification device provided according to an embodiment of the present invention, such as... Figure 5 As shown, the device includes: a data acquisition module 51, a data transmission module 52, a first parsing module 53, a second parsing module 54, and a verification module 55. The device will be described below.

[0079] The acquisition module 51 is used to acquire the first incremental data of the source database and the second incremental data of the target database. The source database is the database of the centralized system, and the target database is the database of the distributed system. The first incremental data is the data that changes in the source database during system migration, and the second incremental data is the data that changes in the target database during system migration.

[0080] The sending module 52, connected to the acquisition module 51, is used to send data matching the first identifier information included in the first incremental data to the first processing thread, and to send data matching the second identifier information included in the second incremental data to the second processing thread.

[0081] The first parsing module 53 is connected to the sending module 52. In the first processing thread, it records and parses the first incremental data to obtain multiple first operation types corresponding to the source database and the first information fields corresponding to each of the multiple first operation types.

[0082] The second parsing module 54, connected to the first parsing module 53, is used to record and parse the second incremental data in the second processing thread to obtain multiple second operation types corresponding to the target database and the second information fields corresponding to each of the multiple second operation types.

[0083] The verification module 55 is connected to the second parsing module 54 and is used to verify the first incremental data and the second incremental data based on the first operation type, the second operation type, the first information field and the second information field to obtain the data verification result, wherein the data verification result includes data matching success, data matching failure and unconfirmed.

[0084] It should be noted that the aforementioned acquisition module 51, transmission module 52, first parsing module 53, second parsing module 54, and verification module 55 correspond to steps S201 to S205 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the aforementioned modules, as part of the device, can run on the computer terminal 10 provided in the embodiments.

[0085] Embodiments of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.

[0086] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data verification method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned data verification method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0087] The processor can invoke information and application programs stored in memory via a transmission device to perform the following steps: Collecting first incremental data from a source database and second incremental data from a target database, wherein the source database is a database of a centralized system, and the target database is a database of a distributed system; the first incremental data represents data that changed in the source database during system migration, and the second incremental data represents data that changed in the target database during system migration; sending data matching the first identifier information included in the first incremental data to a first processing thread, and sending data matching the second identifier information included in the second incremental data to a second processing thread; in the first processing thread, recording and parsing the first incremental data to obtain multiple first operation types corresponding to the source database and first information fields corresponding to each of the multiple first operation types; in the second processing thread, recording and parsing the second incremental data to obtain multiple second operation types corresponding to the target database and second information fields corresponding to each of the multiple second operation types; and verifying the first incremental data and the second incremental data based on the first operation type, the second operation type, the first information field, and the second information field to obtain a data verification result, wherein the data verification result includes data matching success, data matching failure, and unconfirmed.

[0088] Optionally, the processor may also execute program code that performs the following steps: sending data matching the first identifier information included in the first incremental data to the first processing thread, including: constructing multiple first message partitions; delivering the first incremental data to the multiple first message partitions respectively, wherein data corresponding to the same first identifier information is delivered to the same first message partition; extracting data from the multiple first message partitions respectively to construct a first data table to be processed; calculating the first hash value of the data in the first data table to be processed according to the database name of the source database and the first identifier information; and sending the data in the first data table to be processed to the first processing thread based on the first hash value.

[0089] Optionally, the processor may also execute program code that performs the following steps: sending data matching the second identifier information included in the second incremental data to the second processing thread, including: constructing multiple second message partitions; delivering the second incremental data to the multiple second message partitions respectively, wherein data corresponding to the same second identifier information is delivered to the same second message partition; extracting data from the multiple second message partitions respectively to construct a second data table to be processed; calculating the second hash value of the data in the second data table to be processed according to the database name of the target database and the second identifier information; and sending the data in the second data table to be processed to the second processing thread based on the second hash value.

[0090] Optionally, the processor may also execute program code that performs the following steps: In the first processing thread, the first incremental data is recorded and parsed to obtain multiple first operation types corresponding to the source database and the first information fields corresponding to each of the multiple first operation types, including: recording and parsing the first incremental data to obtain multiple first operation types and first information fields, wherein the first operation type includes an add type, an update type, or a delete type, the first information fields corresponding to the add type and the update type are subsequent information fields, and the first information field corresponding to the delete type is a preceding information field.

[0091] Optionally, the processor may also execute program code with the following steps: In the second processing thread, the second incremental data is recorded and parsed to obtain multiple second operation types corresponding to the target database and the second information fields corresponding to each of the multiple second operation types, including: after obtaining the first information field at a preset time interval, the second incremental data is recorded and parsed to obtain multiple second operation types and second information fields, wherein the second operation types include add type, update type or delete type, the second information fields corresponding to add type and update type are subsequent information fields, and the second information fields corresponding to delete type are preceding information fields.

[0092] Optionally, the processor may also execute program code that performs the following steps: based on a first operation type, a first information field of a second operation type, and a second information field, verifying the first incremental data and the second incremental data to obtain a data verification result, including: determining the operation type that matches one of the multiple first operation types among the multiple second operation types, and respectively using it as the first target operation type and the second target operation type; calculating the hash value of the data corresponding to the first target operation type to obtain a third hash value; calculating the hash value of the data corresponding to the second target operation type to obtain a fourth hash value; comparing whether the third hash value and the fourth hash value match; if the third hash value and the fourth hash value do not match, determining that the data verification result is a data matching failure, and comparing the first information field and the second information field to obtain a field difference result; and recording the field difference result.

[0093] Optionally, the processor may also execute program code that includes the following steps: if the data verification result is a data matching failure, obtaining the correspondence between the operation type and the business identification code; determining the target business identification code corresponding to the second incremental data based on the correspondence; and sending a warning message to the user corresponding to the target business identification code based on the level of the target business identification code, wherein the warning message includes a warning level.

[0094] This invention provides a data verification scheme. It involves collecting first incremental data from a source database and second incremental data from a target database. The source database is a centralized system database, and the target database is a distributed system database. The first incremental data represents data that changed in the source database during system migration, and the second incremental data represents data that changed in the target database during system migration. Data matching the first identifier information included in the first incremental data is sent to a first processing thread, and data matching the second identifier information included in the second incremental data is sent to a second processing thread. In the first processing thread, the first incremental data is recorded and parsed to obtain multiple first operation types corresponding to the source database and first information fields corresponding to each of the multiple first operation types. In the second processing thread, the second incremental data is recorded and parsed to obtain multiple second operation types corresponding to the target database and the second information fields corresponding to each of the multiple second operation types. Based on the first operation type, the second operation type, the first information field, and the second information field, the first incremental data and the second incremental data are checked to obtain the data check result. The data check result includes data matching success, data matching failure, and unconfirmed. This achieves the goal of improving real-time monitoring capabilities and ensuring data consistency in a distributed system migration environment, thereby improving the efficiency and accuracy of data check and solving the technical problem of low efficiency and insufficient accuracy of dual-system data check in the current distributed system migration process.

[0095] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a non-volatile storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0096] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the aforementioned non-volatile storage medium can be used to store the program code executed by the data verification method provided in the above embodiments.

[0097] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0098] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: collecting first incremental data from a source database and second incremental data from a target database, wherein the source database is a database of a centralized system, the target database is a database of a distributed system, the first incremental data is data that changes in the source database during system migration, and the second incremental data is data that changes in the target database during system migration; sending data matching the first identifier information included in the first incremental data to a first processing thread, and sending data matching the second identifier information included in the second incremental data to a second processing thread; in the first processing thread, recording and parsing the first incremental data to obtain multiple first operation types corresponding to the source database and first information fields corresponding to each of the multiple first operation types; in the second processing thread, recording and parsing the second incremental data to obtain multiple second operation types corresponding to the target database and second information fields corresponding to each of the multiple second operation types; and verifying the first incremental data and the second incremental data based on the first operation type, the second operation type, the first information field, and the second information field to obtain a data verification result, wherein the data verification result includes data matching success, data matching failure, and unconfirmed.

[0099] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: sending data matching the first identifier information included in the first incremental data to the first processing thread, including: constructing multiple first message partitions; delivering the first incremental data to the multiple first message partitions respectively, wherein data corresponding to the same first identifier information is delivered to the same first message partition; extracting data from the multiple first message partitions respectively to construct a first data table to be processed; calculating the first hash value of the data in the first data table to be processed according to the database name of the source database and the first identifier information; and sending the data in the first data table to be processed to the first processing thread based on the first hash value.

[0100] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: sending data matching the second identifier information included in the second incremental data to the second processing thread, including: constructing multiple second message partitions; delivering the second incremental data to the multiple second message partitions respectively, wherein data corresponding to the same second identifier information is delivered to the same second message partition; extracting data from the multiple second message partitions respectively to construct a second data table to be processed; calculating the second hash value of the data in the second data table to be processed according to the database name of the target database and the second identifier information; and sending the data in the second data table to be processed to the second processing thread based on the second hash value.

[0101] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: In the first processing thread, the first incremental data is recorded and parsed to obtain multiple first operation types corresponding to the source database and first information fields corresponding to each of the multiple first operation types, including: recording and parsing the first incremental data to obtain multiple first operation types and first information fields, wherein the first operation type includes an add type, an update type, or a delete type, the first information fields corresponding to the add type and the update type are post-information fields, and the first information field corresponding to the delete type is a pre-information field.

[0102] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: In the second processing thread, the second incremental data is recorded and parsed to obtain multiple second operation types corresponding to the target database and the second information fields corresponding to each of the multiple second operation types, including: after obtaining the first information field, the second incremental data is recorded and parsed to obtain multiple second operation types and second information fields, wherein the second operation types include add type, update type or delete type, the second information fields corresponding to add type and update type are subsequent information fields, and the second information fields corresponding to delete type are preceding information fields.

[0103] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: Based on a first operation type, a first information field of a second operation type, and a second information field, verifying the first incremental data and the second incremental data to obtain a data verification result, including: determining an operation type among multiple first operation types that matches one of multiple second operation types, respectively as a first target operation type and a second target operation type; calculating the hash value of the data corresponding to the first target operation type to obtain a third hash value; calculating the hash value of the data corresponding to the second target operation type to obtain a fourth hash value; comparing whether the third hash value and the fourth hash value match; if the third hash value and the fourth hash value do not match, determining that the data verification result is a data matching failure, and comparing the first information field and the second information field to obtain a field difference result; and recording the field difference result.

[0104] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: further including: in the case that the data verification result is a data matching failure, obtaining the correspondence between the operation type and the service identification code; based on the correspondence, determining the target service identification code corresponding to the second incremental data; and based on the level of the target service identification code, sending a warning message to the user corresponding to the target service identification code, wherein the warning message includes a warning level.

[0105] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0106] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0108] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0109] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0110] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0111] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A data collation method, characterized by, The application comprises: collecting first incremental data of a source database and second incremental data of a target database, wherein the source database is a database of a centralized system, the target database is a database of a distributed system, the first incremental data is data changed in the source database during system migration, and the second incremental data is data changed in the target database during the system migration; sending data matched with first identifier information included in the first incremental data to a first processing thread, and sending data matched with second identifier information included in the second incremental data to a second processing thread; recording and analyzing the first incremental data in the first processing thread to obtain a plurality of first operation types corresponding to the source database and first information fields corresponding to the plurality of first operation types respectively; recording and analyzing the second incremental data in the second processing thread to obtain a plurality of second operation types corresponding to the target database and second information fields corresponding to the plurality of second operation types respectively; checking the first incremental data and the second incremental data based on the first operation types, the second operation types, the first information fields and the second information fields to obtain a data checking result, wherein the data checking result comprises data matching success, data matching failure and unconfirmed.

2. The method of claim 1, wherein, The sending of data matched with first identifier information included in the first incremental data to a first processing thread comprises: constructing a plurality of first message partitions; delivering the first incremental data into the plurality of first message partitions respectively, wherein data corresponding to the same first identifier information is delivered into the same first message partition; extracting data from the plurality of first message partitions respectively to construct a first to-be-processed data table; calculating first hash values of data in the first to-be-processed data table according to a database name of the source database and the first identifier information; sending the data in the first to-be-processed data table to the first processing thread based on the first hash values.

3. The method of claim 1, wherein, The sending of data matched with second identifier information included in the second incremental data to a second processing thread comprises: constructing a plurality of second message partitions; delivering the second incremental data into the plurality of second message partitions respectively, wherein data corresponding to the same second identifier information is delivered into the same second message partition; extracting data from the plurality of second message partitions respectively to construct a second to-be-processed data table; calculating second hash values of data in the second to-be-processed data table according to a database name of the target database and the second identifier information; sending the data in the second to-be-processed data table to the second processing thread based on the second hash values.

4. The method of claim 1, wherein, The recording and analyzing of the first incremental data in the first processing thread to obtain a plurality of first operation types corresponding to the source database and first information fields corresponding to the plurality of first operation types respectively comprises: The first incremental data is recorded and analyzed to obtain the plurality of first operation types and the first information fields, wherein the first operation types include an addition type, an update type, or a deletion type, the first information fields corresponding to the addition type and the update type are post information fields, and the first information field corresponding to the deletion type is a pre information field.

5. The method of claim 1, wherein, The second incremental data is recorded and analyzed in the second processing thread to obtain a plurality of second operation types corresponding to the target database and second information fields corresponding to the plurality of second operation types, including: The second incremental data is recorded and analyzed to obtain the plurality of second operation types and the second information fields after a preset time interval of obtaining the first information fields, wherein the second operation types include an addition type, an update type, or a deletion type, the second information fields corresponding to the addition type and the update type are post information fields, and the second information field corresponding to the deletion type is a pre information field.

6. The method of claim 1, wherein, The first incremental data and the second incremental data are checked based on the first operation types, the second operation types, the first information fields, and the second information fields to obtain a data checking result, including: determining operation types in the plurality of first operation types that match operation types in the plurality of second operation types as first target operation types and second target operation types, respectively; calculating a hash value of data corresponding to the first target operation types to obtain a third hash value; calculating a hash value of data corresponding to the second target operation types to obtain a fourth hash value; comparing whether the third hash value and the fourth hash value match; in a case where the third hash value and the fourth hash value do not match, determining that the data checking result is a data matching failure, and comparing the first information fields and the second information fields to obtain a field difference result; recording the field difference result.

7. The method according to any one of claims 1 to 6, characterized in that, Further comprising: in a case where the data checking result is the data matching failure, obtaining a correspondence between operation types and business identification codes; based on the correspondence, determining a target business identification code corresponding to the second incremental data; based on a level of the target business identification code, sending warning information to a user corresponding to the target business identification code, wherein the warning information includes a warning level.

8. A data collating apparatus, characterized by comprising: comprising: a collection module configured to collect first incremental data of a source database and second incremental data of a target database, wherein the source database is a database of a centralized system, the target database is a database of a distributed system, the first incremental data is data that has changed in the source database during system migration, and the second incremental data is data that has changed in the target database during the system migration; a sending module configured to send data including a first identifier information matching the first incremental data to a first processing thread and send data including a second identifier information matching the second incremental data to a second processing thread; and The first analysis module records and analyzes the first incremental data in the first processing thread to obtain a plurality of first operation types corresponding to the source database, and a first information field corresponding to each of the plurality of first operation types; The second analysis module records and analyzes the second incremental data in the second processing thread to obtain a plurality of second operation types corresponding to the target database, and a second information field corresponding to each of the plurality of second operation types; The checking module checks the first incremental data and the second incremental data based on the first operation types, the second operation types, the first information field, and the second information field, to obtain a data checking result, wherein the data checking result includes data matching success, data matching failure, and unconfirmed.

9. A non-volatile storage medium, comprising: The non-volatile storage medium includes a stored program, wherein the program controls the device in which the non-volatile storage medium is located to execute the data checking method of any one of claims 1 to 7 when the program is running.

10. A computer device, comprising: Comprise: Memory and processor, The memory stores a computer program; The processor is configured to execute the computer program stored in the memory, and the computer program causes the processor to execute the data checking method of any one of claims 1 to 7 when the computer program is running.

Images