A data processing method, apparatus, device, and medium

Abstract

The application provides a data processing method, comprising: comparing first full data from a source database and second full data from a target database, the first full data and the second full data being data generated before triggering a data comparison operation; analyzing a transaction log of the source database to obtain first incremental data, the first incremental data being data generated after triggering the data comparison operation; and querying whether the first incremental data exists in the target database. Thus, comparison of database data can be performed without stopping business writing, normal operation of the business is ensured, and user experience is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the advent of the information age, massive amounts of data have been generated. To ensure the normal operation of businesses, many enterprises, individuals, and organizations need to build data centers to store this data, thus increasing operating costs. Based on this, the industry has provided a cloud service. Specifically, cloud service providers build data centers and then provide storage resources to enterprises and other organizations through these data centers. These organizations can then migrate data stored in their local data centers to the cloud. This reduces the costs of operating and maintaining data centers.

[0003] However, when enterprises and other organizations migrate data from on-premises to the cloud, they typically query all data from the on-premises source database and all data from the cloud target database, then compare the data retrieved from the source database and the target database offline. To ensure the accuracy of the comparison, business writes usually need to be stopped, which impacts the user experience.

[0004] Therefore, the industry urgently needs to provide a data processing method to ensure normal business operations and avoid affecting user experience. Summary of the Invention

[0005] This application provides a data processing method that enables accurate data comparison between different databases without interrupting business write operations, ensuring normal business operation and improving user experience. This application also provides corresponding apparatus, devices, computer-readable storage media, and computer program products.

[0006] Firstly, this application provides a data processing method applicable to scenarios involving database comparison, including database migration, database synchronization, and database disaster recovery. In its implementation, it first compares a first full dataset from the source database with a second full dataset from the target database. The first and second full datasets are data generated before the data comparison operation is triggered. Next, it parses the transaction logs of the source database to obtain first incremental data, which is data generated after the data comparison operation is triggered. Then, it queries whether this first incremental data exists in the target database. This allows for database data comparison without stopping business writes, ensuring normal business operation and improving user experience.

[0007] In conjunction with the first aspect, in the first implementation of the first aspect, when there is inconsistency between the first full data and the second full data, the inconsistent data between the first full data and the second full data can be re-checked. This can improve the accuracy of data comparison in the database.

[0008] In conjunction with the first implementation of the first aspect, in the second implementation of the first aspect, when the re-inspection result indicates that the inconsistent data remains inconsistent after re-inspection—that is, when the data from the source database and the target database added to the re-inspection queue are inconsistent—it can be determined whether the inconsistent data has changed relative to the previous comparison operation. If it has not changed, it indicates that the data in the source database and the data in the target database are indeed inconsistent, and the comparison result can be reported so that the target database can be updated based on the comparison result.

[0009] In conjunction with the second implementation of the first aspect, in the third implementation of the first aspect, if the inconsistent data changes relative to the previous comparison operation, it indicates that the data is still in a state of change. The inconsistent data can be checked again until the inconsistent data is still inconsistent after re-checking, and the inconsistent data no longer changes.

[0010] By combining the first aspect or any of the first to third implementations of the first aspect, when comparing the first full data from the source database and the second full data from the target database, the first full data can be divided into N first fragments and the second full data into N second fragments according to a preset rule, where N is a positive integer. The first fragments and the second fragments are in one-to-one correspondence. Then, the N first fragments and the N second fragments are compared concurrently. This improves comparison efficiency, allowing users to obtain data consistency results in real time, thus meeting the needs of real-time business operations.

[0011] Secondly, this application provides a data processing apparatus applied to scenarios involving database comparison, including database migration, database synchronization, and database disaster recovery. Specifically, the apparatus includes a comparison module, a parsing module, and a query module. The comparison module compares a first full dataset from a source database with a second full dataset from a target database, wherein the first and second full datasets are data generated before triggering the data comparison operation. The parsing module parses the transaction logs of the source database to obtain first incremental data, wherein the first incremental data is data generated after triggering the data comparison operation, and this first incremental data refers to data in the source database. The query module queries whether the first incremental data exists in the target database.

[0012] This device performs data comparison in two parts: one before the data comparison operation is triggered and another after the data comparison operation is triggered. Specifically, it compares the first full data and the second full data, as well as the first incremental data and the data in the target database. On the one hand, it enables accurate data comparison between different databases without stopping business writes, ensuring normal business operation and improving user experience. On the other hand, it allows subsequent comparisons to focus mainly on incremental data, thus improving comparison efficiency, providing real-time feedback on comparison results, and enabling immediate detection of data consistency issues, thereby meeting business needs.

[0013] In conjunction with the second aspect, in the first implementation of the second aspect, the comparison module is further used for:

[0014] When there is a data inconsistency between the first full data and the second full data, the inconsistent data between the first full data and the second full data is re-examined.

[0015] In conjunction with the first implementation of the second aspect, in the second implementation of the second aspect, the device further includes: a judgment module, configured to determine whether the inconsistent data has changed relative to the previous comparison operation when the re-inspection result indicates that the inconsistent data is still inconsistent after re-inspection; and a communication module, configured to report the comparison result if not, so as to update the target database according to the comparison result.

[0016] In conjunction with the second implementation of the second aspect, in the third implementation of the second aspect, the comparison module is further configured to: if so, re-examine the inconsistent data until the inconsistent data is still inconsistent after re-examination, and the inconsistent data no longer changes.

[0017] In conjunction with the second aspect or any of the first to third implementations of the second aspect, the comparison module is specifically used to: divide the first full data into N first fragments and the second full data into N second fragments according to a preset rule, where N is a positive integer, and the first fragments and the second fragments correspond one-to-one; and compare the N first fragments and the N second fragments in a concurrent manner.

[0018] Thirdly, this application provides an apparatus comprising a processor and a memory;

[0019] The processor is configured to execute instructions stored in the memory to cause the device to perform the data processing method as described in the first aspect.

[0020] Fourthly, this application provides a computer-readable storage medium including instructions that, when executed on a device, cause the device to perform the data processing method as described in the first aspect.

[0021] Fifthly, this application provides a computer program product containing instructions that, when run on a computer cluster, causes the computer cluster to execute the data processing method described in the first aspect or any implementation thereof.

[0022] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0023] To more clearly illustrate the technical methods of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly described below.

[0024] Figure 1 A system architecture diagram of a data processing method provided in an embodiment of this application;

[0025] Figure 2 An interactive flowchart of a data processing method provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of the structure of a device provided in an embodiment of this application. Detailed Implementation

[0027] The solutions in the embodiments provided in this application will now be described with reference to the accompanying drawings.

[0028] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application.

[0029] To facilitate understanding of the technical solution of this application, some technical terms involved in this application will be introduced below.

[0030] A database, simply put, can be viewed as an electronic filing cabinet. A database is a place to store electronic files, through which users can perform operations such as adding, retrieving, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A database consists of multiple tablespaces.

[0031] A tablespace is a logical partition of a database, especially a relational database. A tablespace can only belong to one database. All database objects are stored in a designated tablespace. However, the primary storage is for tables, hence the name "tablespace." Each tablespace has a unique identifier, denoted as a tablespace ID.

[0032] Databases are maintained based on transactions. A transaction is the unit for maintaining the consistency and integrity of database data, possessing the characteristic of "either all succeed or all fail." Before a transaction is committed, the database is not responsible for the consistency and integrity of its data. However, once a transaction is committed, the database must guarantee the consistency and integrity of the data. Simply put, once a transaction is committed, no data can be lost. Some databases (such as relational databases), in order to improve performance, generally consider a commit complete after writing transaction data to the transaction log and buffer pool, rather than considering it complete only after writing it to the page file.

[0033] The transaction log (also known as the redo log) is a file stored directly on persistent storage devices such as disks or solid-state drives (SSDs). Its data structure is simpler than that of a page file, and it uses sequential storage. A transaction is committed when it is written to the transaction log and the buffer pool. However, the "flush" process—writing dirty pages (memory data pages whose contents differ from their corresponding disk data pages) to the disk or SSD—is asynchronous. If an exception occurs, such as a crash, while some dirty pages remain unwritten to the disk or SSD, the data in the buffer pool will be lost. In this case, the transaction log is needed to restore the data.

[0034] Based on this, all data can be written to the log file (or log system) in a specific format according to the transaction execution order before being written. This follows the write-ahead logging (WAL) principle. The logs written to the log file (or log system) include the log sequence number (LSN). The LSN is an incrementing integer representing the total number of bytes written to the transaction log. Because it increments continuously, the LSN serves as an identifier, identifying transactions written at different times. When a database exception occurs, such as a crash, recovery can be performed based on the LSN. It should be noted that the LSN is stored not only in the transaction log but also in the buffer pool.

[0035] Many current business scenarios involve writing data from one database to another. For example, an increasing number of enterprises are choosing to migrate their business to the cloud, i.e., migrating business data to the cloud to reduce local maintenance costs. Another example is when an enterprise expands, it can replicate business data to a new database to serve more users. Alternatively, in security scenarios, data may be written to one or more other databases to achieve disaster recovery. The database being written to is called the target database, and the database from which the data is written is called the source database.

[0036] When writing data from the source database to the target database, the data in both databases is typically compared first. Specifically, all data is retrieved from both the source and target databases, and then the data retrieved from the source and target databases is compared offline. To ensure accuracy, business operations are usually stopped, which impacts user experience.

[0037] To address this, this application provides a data processing method that divides data comparison into two parts. Specifically, it first compares a first full dataset from the source database and a second full dataset from the target database, where the first and second full datasets are data generated before the data comparison operation is triggered. Next, it parses the transaction log of the source database to obtain first incremental data, which is data generated after the data comparison operation is triggered. Then, it queries whether this first incremental data exists in the target database. This allows for database data comparison without stopping business writes, ensuring normal business operation and improving user experience. Furthermore, subsequent comparisons mainly focus on the incremental data, thus improving comparison efficiency, providing real-time feedback on comparison results, and enabling immediate detection of data consistency issues, meeting business requirements.

[0038] The data processing method provided in this application can be executed by a data processing device. The aforementioned data processing method may be applied to, but is not limited to, [various applications]. Figure 1 The application scenarios shown.

[0039] like Figure 1As shown, this application scenario includes a data processing device 100, a source database 12, and a target database 14. The data processing device 100 includes a communication module 102, a comparison module 104, a parsing module 106, and a query module 108. The communication module 102 is specifically used to obtain first full data from the source database 12 and second full data from the target database 14. The first and second full data are data generated before triggering the data comparison operation. The comparison module 104 is used to compare the first and second full data to determine data consistency. The communication module 102 is also used to obtain the transaction log of the source database 12. The parsing module 106 is used to parse the transaction log of the source database 12 to obtain first incremental data, which is data generated after triggering the data comparison operation. The query module 108 is specifically used to query whether the first incremental data exists in the target database.

[0040] The data processing device 100 can be deployed in a device with data processing capabilities. This device can be a terminal or a server. Terminals include desktop computers, laptops, smartphones, etc., while servers include dedicated servers or cloud servers. The source database 12 and the target database 14 can be deployed in a device with data storage capabilities. This device can be a terminal or a server. It should be noted that the source database 12 and the target database 14 can be deployed in the same device or in different devices. In some cases, the data processing device 100 can also be deployed in the same device as either the source database 12 or the target database. Of course, deploying the data processing device 100, the source database 12, and the target database 14 in different devices improves the robustness of the entire system.

[0041] To make the technical solution of this application clearer and easier to understand, the data processing method provided in this application will be described from the perspective of the data processing device 100 below.

[0042] See Figure 2 The flowchart shown illustrates a data processing method, which includes:

[0043] S202: The data processing device 100 obtains the first full data from the source database 12.

[0044] S204: The data processing device 100 obtains the second full data from the target database 14.

[0045] The first full data and the second full data refer to the data generated before the data comparison operation is triggered. The first full data is stored in the source database 12, and the second full data is stored in the target database 14. For ease of understanding, the first full data and the second full data are illustrated below with specific examples.

[0046] In one example, if a user triggers a comparison operation at 14:13:20 on February 17, 2020, then the data written to the source database 12 before 14:13:20 on February 17, 2020 is the first full data, and the data written to the target database 14 before 14:13:20 on February 17, 2020 is the second full data.

[0047] In specific implementation, the data processing device 100 can obtain the first full data through the data access interface provided by the source database 12, and obtain the second full data through the data access interface provided by the target database 14. It should be noted that the data processing device 100 can obtain the aforementioned full data from both the source database 12 and the target database 14 simultaneously, or it can obtain the first full data and the second full data sequentially according to a set execution order. That is, S202 and S204 can be executed in parallel, or they can be executed sequentially according to a set order.

[0048] S206: The data processing device 100 compares the first full data and the second full data. If there is a data inconsistency between the first full data and the second full data, S208 is executed.

[0049] Data in a database is typically stored in key-value pairs. In other words, the first and second full datasets exist as key-value pairs. Based on this, the data processing device 100 can compare the first and second full datasets by comparing the values ​​corresponding to the same key.

[0050] To facilitate understanding, this application also provides a specific example for illustration.

[0051] Assuming key-value pairs represent the traffic of a certain port, the first full dataset includes data for port A as <traffic, 3Mb / s>, meaning the traffic of port A is 3M. ​​The second full dataset includes data for port A as <traffic, 2Mb / s>. The data processing device 100 determines the consistency of the data in the first and second full datasets by comparing the key-value pairs. In this example, the data is clearly inconsistent. The data processing device 100 can add this set of data to a review queue and then compare it with the next set of data.

[0052] In some implementations, the data processing device 100 can divide the first full data into N first fragments and the second full data into N second fragments according to preset rules. Here, N is a positive integer, and the first fragments and second fragments correspond one-to-one. In this way, the data processing device 100 can compare the N first fragments and the N second fragments concurrently. This improves comparison efficiency, allowing users to obtain data consistency results in real time, thus meeting the needs of real-time services.

[0053] The aforementioned preset rules can be based on the amount of data or the number of shards. For example, the size of each shard can be set, or the number of shards can be set. Based on this, the preset rules could be that the shard size does not exceed 1000M, or the number of shards does not exceed 1000.

[0054] S208: The data processing device 100 re-examines the inconsistent data in the first full data and the second full data. If the re-examination result indicates that the inconsistent data are still inconsistent after re-examination, S210 is executed.

[0055] Specifically, the data processing device 100 can read data from the re-inspection queue, obtain inconsistent data between the first full data and the second full data, and re-inspect that data. During the re-inspection, the data processing device 100 can re-inspect each piece of data one by one.

[0056] When the amount of data to be re-examined is large, the data processing device 100 can also divide the data in the re-examined queue into multiple segments. For example, the data from the source database 12 in the re-examined queue can be divided into m first segments, and the data from the target database 14 in the re-examined queue can be divided into m second segments, where m is a positive integer. Then, the m first segments and the m second segments are compared separately to improve the re-examined efficiency.

[0057] When performing a re-inspection, if the data is inconsistent, S210 can be executed to determine whether the data has changed. If the data is consistent, the re-inspection and subsequent judgment operations can be omitted.

[0058] S210: The data processing device 100 determines whether the inconsistent data has changed relative to the previous comparison operation. If not, it executes S212. If yes, it returns to execute S208.

[0059] For data that is inconsistent during re-inspection, the data processing device 100 compares key-value pairs to determine whether the data has changed compared to the previous comparison operation. Specifically, the data processing device 100 can compare the key-value pairs from the source database 12 during the re-inspection with the decrement pairs from the source database 12 during the previous comparison to determine whether the values ​​corresponding to the same keys have changed.

[0060] To facilitate understanding, a specific example is provided below. In one example, the data for port A in the re-examination queue is <flow rate, 4Mb / s>, indicating that the flow rate of port A during the re-examination is 4Mb / s. The data for port A during the previous comparison is <flow rate, 3Mb / s>, indicating that the flow rate of port A during the previous comparison is 3Mb / s. Therefore, it can be determined that the data for port A has changed. Thus, the data processing device 100 can return to S208 to perform another re-examination. If the re-examination is still inconsistent, it will determine whether the inconsistent data has changed relative to the previous comparison operation. When the inconsistent data no longer changes relative to the previous comparison operation, it indicates that the data is indeed inconsistent, and the data processing device 100 can execute S212 to report the comparison result.

[0061] S212: Data processing device 100 reports the comparison results.

[0062] The comparison results include, at a minimum, the data from the source database 12 when the inconsistent data has not changed since the last comparison operation. Of course, in some cases, the comparison results may also include the aforementioned data from the target database 14 corresponding to the source database 12.

[0063] To facilitate understanding, a specific example is provided below. In one example, when the inconsistent data has not changed since the last comparison operation, the data from source database 12 includes <traffic, 5Mb / s>, and the data from target database 14 includes <traffic, 2Mb / s>. Based on this, the comparison result can include data from source database 12, i.e., <traffic, 5Mb / s>. Of course, in some cases, the comparison result can also include data from target database 14, i.e., "<traffic, 5Mb / s>, <traffic, 2Mb / s>". Here, the left side of the comma represents data from source database 12, and the right side represents data from target database 14.

[0064] S214: Target database 14 updates data based on the comparison results.

[0065] In practice, the target database 14 updates the corresponding data in itself based on the data from the source database 12 in the comparison results. Furthermore, to ensure accuracy, when the comparison results include data from the target database 14, the target database 14 can first verify the corresponding data in itself, that is, compare whether the data from the target database 14 in the comparison results is consistent with the corresponding data stored in the target database 14. If they are consistent, the verification passes, and then the corresponding data in the target database 14 is updated based on the data from the source database 12 in the comparison results. If they are inconsistent, the verification fails, and the target database 14 can search for the corresponding data again or re-obtain the comparison results until the verification passes.

[0066] S216: The data processing device 100 acquires the transaction log of the source database 12.

[0067] S218: The data processing device 100 parses the transaction log and obtains the first incremental data.

[0068] Specifically, the data processing device 100 can obtain the transaction log of the source database 12. Based on the log sequence number (LSN) of the transaction log (specifically, the log sequence number at the current moment and the sequence number when the comparison operation was triggered), the data processing device 100 can determine the database operations that occurred within the time period after the comparison operation was triggered. These database operations can include any one or more of adding, modifying, or deleting data. By parsing the operation records of the aforementioned database operations in the transaction log, the data processing device 100 can obtain the first incremental data. This first incremental data is the data generated after the comparison operation was triggered, specifically the data written to the source database 12 after the comparison operation was triggered.

[0069] S220: The data processing device 100 queries whether the first incremental data exists in the target database 14.

[0070] In practical applications, the data processing device 100 can acquire the data written to the target database 14 after the comparison operation is triggered, and then determine whether the first incremental data exists in the target database 14 by comparing the first incremental data with the data written to the target database 14 after the comparison operation is triggered.

[0071] It should be noted that S202 to S214 above are specific implementations for comparing the full set of data, and S216 to S220 above are specific implementations for comparing the incremental data. The two can be executed in parallel or sequentially according to a set order. Furthermore, S208 to S214 above are optional implementations provided in this application's embodiments. In other possible implementations of this application's embodiments, they can also be executed in other ways, or not at all.

[0072] Based on the above description, this application provides a data processing method that first compares a first full dataset from a source database with a second full dataset from a target database. Next, it parses the transaction log of the source database to obtain first incremental data, which is specifically the data generated after triggering the data comparison operation. Then, it queries whether this first incremental data exists in the target database. This allows for database data comparison without stopping business writes, ensuring normal business operation and improving user experience.

[0073] Based on the above implementation of the data processing method provided in the embodiments of this application, the embodiments of this application also provide corresponding devices and equipment, which will be described in detail below.

[0074] See Figure 1 The diagram shows the structure of the data processing device 100, which includes a communication module 102, a comparison module 104, a parsing module 106, and a query module 108.

[0075] The communication module 102 is used to acquire first full data from the source database 12 and second full data from the target database 14, wherein the first full data and the second full data are data generated before triggering the data comparison operation. The comparison module 104 is specifically used to compare the first full data from the source database 12 and the second full data from the target database 14.

[0076] The communication module 102 is also used to acquire the transaction logs of the source database 12. Correspondingly, the parsing module 106 is used to parse the transaction logs of the source database 12 to obtain the first incremental data. The first incremental data is the data generated after triggering the data comparison operation. The query module 108 is used to query whether the first incremental data exists in the target database.

[0077] For a detailed implementation of the communication module 102, please refer to [link / reference needed]. Figure 2 The embodiments shown describe the relevant content of S202 to S204. For a detailed implementation of the comparison module 104, please refer to... Figure 2The embodiment shown describes the relevant content of S206. For a more detailed implementation of the communication module 102, please refer to... Figure 2 The embodiment shown describes the relevant content of S216. For a detailed implementation of the parsing module 106, please refer to... Figure 2 The embodiment shown describes the relevant content of S218. For a detailed implementation of the query module 108, please refer to... Figure 2 Description of S220 in the illustrated embodiment.

[0078] In some possible implementations, the comparison module 104 is further configured to:

[0079] When there is a data inconsistency between the first full data and the second full data, the inconsistent data between the first full data and the second full data is re-examined.

[0080] For a detailed implementation of the comparison module 104, please refer to [link / reference]. Figure 2 Description of S208 in the illustrated embodiment.

[0081] In some possible implementations, the device 100 further includes: a judgment module, configured to determine whether the inconsistent data has changed relative to the previous comparison operation when the re-inspection result indicates that the inconsistent data is still inconsistent after re-inspection; and a communication module 102, configured to report the comparison result if not, so as to update the target database according to the comparison result.

[0082] For a detailed implementation of the judgment module, please refer to [link / reference]. Figure 2 The description of S210 in the illustrated embodiment is provided. For a more detailed implementation of the communication module 102, please refer to... Figure 2 Description of S212 in the illustrated embodiment.

[0083] In some possible implementations, the comparison module 104 is further configured to: if so, re-examine the inconsistent data until the inconsistent data is still inconsistent after re-examination, and the inconsistent data no longer changes.

[0084] In some possible implementations, the comparison module 104 is specifically used for:

[0085] According to preset rules, the first full data is divided into N first fragments, and the second full data is divided into N second fragments, where N is a positive integer. The first fragments and the second fragments correspond one-to-one. The N first fragments and the N second fragments are compared in a concurrent manner.

[0086] For the specific implementation of the comparison module 104, please refer to [link / reference]. Figure 2 Description of S206 in the illustrated embodiment.

[0087] The data processing apparatus 100 according to the embodiments of this application can correspondingly execute the data processing method described in the embodiments of this application, and the above and other operations and / or functions of each module in the data processing apparatus 100 are respectively for implementing Figure 2 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0088] Figure 3 A device is also provided. Figure 3 The device 300 shown can be specifically used to achieve the above. Figure 1 The data processing device 100 in the illustrated embodiment has the following functions.

[0089] Device 300 includes a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, memory 304, and communication interface 303 communicate via bus 301. Bus 301 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The symbol is represented by a single thick line, but this does not mean that there is only one bus or one type of bus. Communication interface 303 is used to communicate with external systems, such as obtaining the first full data from source database 12, obtaining the second full data from target database 14, or obtaining the transaction log of source database 12, etc.

[0090] The processor 302 may be a central processing unit (CPU). The memory 304 may include volatile memory, such as random access memory (RAM). The memory 304 may also include non-volatile memory, such as read-only memory (ROM), flash memory, HDD, or SSD.

[0091] The memory 304 stores executable code, and the processor 302 executes the executable code to perform the aforementioned code data processing method.

[0092] Specifically, in achieving Figure 1 In the case of the illustrated embodiment, and Figure 1 When the modules described in the embodiments are implemented in software, they execute... Figure 1The software or program code required for the functions of the comparison module 104, parsing module 106, and query module 108 are stored in the memory 304. The function of the communication module 102 is implemented through the communication interface 303. The first full data, the second full data, and the transaction log obtained through the communication interface 303 can be stored in the memory 304. The processor 302 is used to execute the instructions in the memory 304 to perform the data processing method applied to the data processing device 100.

[0093] Specifically, the processor 302 is used to execute instructions in the memory 304, performing the following method steps:

[0094] Compare the first full data from the source database with the second full data from the target database. The first full data and the second full data are data generated before the data comparison operation is triggered.

[0095] The transaction log of the source database is parsed to obtain the first incremental data, which is the data generated after the data comparison operation is triggered;

[0096] Check if the first incremental data exists in the target database.

[0097] Optionally, the processor 302 is also configured to execute instructions in the memory 304, performing the following method steps:

[0098] When there is a data inconsistency between the first full data and the second full data, the inconsistent data between the first full data and the second full data is re-examined.

[0099] Optionally, the processor 302 is also configured to execute instructions in the memory 304, performing the following method steps:

[0100] If the re-inspection result indicates that the inconsistent data is still inconsistent, it is determined whether the inconsistent data has changed relative to the previous comparison operation; if not, the comparison result is reported so that the target database can be updated according to the comparison result.

[0101] Optionally, the processor 302 is also configured to execute instructions in the memory 304, performing the following method steps:

[0102] If so, the inconsistent data will be re-examined until the inconsistent data is still inconsistent after re-examination and the inconsistent data no longer changes.

[0103] Optionally, the processor 302 is used to execute instructions in the memory 304, performing the following method steps:

[0104] According to preset rules, the first full data is divided into N first fragments, and the second full data is divided into N second fragments, where N is a positive integer, and the first fragments and the second fragments correspond one-to-one.

[0105] The N first fragments and the N second fragments are compared in a concurrent manner.

[0106] This application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the data processing method applied to the data processing apparatus 100 described above.

[0107] This application also provides a computer program product, which, when executed by a computer, enables the computer to perform any of the aforementioned data processing methods. The computer program product can be a software installation package; when any of the aforementioned data processing methods needs to be used, the computer program product can be downloaded and executed on the computer.

[0108] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0109] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0110] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0111] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: Compare the first full data from the source database with the second full data from the target database. The first full data and the second full data are data generated before the data comparison operation is triggered. While comparing the first full data and the second full data, the transaction log of the source database is parsed in parallel to obtain the first incremental data, which is the data generated after the data comparison operation is triggered. Check if the first incremental data exists in the target database; The comparison of the first full dataset from the source database and the second full dataset from the target database includes: The first full data is divided into N first fragments according to a preset rule, and the second full data is divided into N second fragments, where N is a positive integer. The first fragments and the second fragments correspond one-to-one. The preset rule is a rule set based on the amount of data or the number of fragments. The N first fragments and the N second fragments are compared concurrently. The step of querying whether the first incremental data exists in the target database includes: Retrieve the data written to the target database after the comparison operation is triggered; By comparing the first incremental data with the data written to the target database after the comparison operation is triggered, it is determined whether the first incremental data exists in the target database.

2. The method according to claim 1, characterized in that, When there is a data inconsistency between the first full data and the second full data, the method further includes: Data that is inconsistent between the first full dataset and the second full dataset is re-examined.

3. The method according to claim 2, characterized in that, When the re-inspection results indicate that the inconsistent data are still inconsistent after re-inspection, the method further includes: Determine whether the inconsistent data has changed relative to the previous comparison operation; If not, the comparison results are reported so that the target database can be updated based on the comparison results.

4. The method according to claim 3, characterized in that, The method further includes: If so, the inconsistent data will be re-examined until the inconsistent data is still inconsistent after re-examination and the inconsistent data no longer changes.

5. A data processing apparatus, characterized in that, The device includes: The comparison module is used to compare the first full data from the source database and the second full data from the target database. The first full data and the second full data are data generated before the data comparison operation is triggered. The parsing module is used to parse the transaction log of the source database in parallel while comparing the first full data and the second full data to obtain the first incremental data, which is the data generated after the data comparison operation is triggered. The query module is used to query whether the first incremental data exists in the target database; The comparison module is specifically used for: The first full data is divided into N first fragments according to a preset rule, and the second full data is divided into N second fragments, where N is a positive integer. The first fragments and the second fragments correspond one-to-one. The preset rule is a rule set based on the amount of data or the number of fragments. The N first fragments and the N second fragments are compared concurrently. The query module is specifically used for: Retrieve the data written to the target database after the comparison operation is triggered; By comparing the first incremental data with the data written to the target database after the comparison operation is triggered, it is determined whether the first incremental data exists in the target database.

6. The apparatus according to claim 5, characterized in that, The comparison module is also used for: When there is a data inconsistency between the first full data and the second full data, the inconsistent data between the first full data and the second full data is re-examined.

7. The apparatus according to claim 6, characterized in that, The device further includes: The judgment module is used to determine whether the inconsistent data has changed relative to the previous comparison operation when the re-inspection result indicates that the inconsistent data is still inconsistent after re-inspection. A communication module is used to report the comparison results if not, so as to update the target database based on the comparison results.

8. The apparatus according to claim 7, characterized in that, The comparison module is also used for: If so, the inconsistent data will be re-examined until the inconsistent data is still inconsistent after re-examination and the inconsistent data no longer changes.

9. A device, characterized in that, The device includes a processor and a memory; The processor is configured to execute instructions stored in the memory to cause the device to perform the data processing method as described in any one of claims 1 to 4.

10. A computer-readable storage medium comprising instructions, when executed on a device, causing the device to perform the data processing method as claimed in any one of claims 1 to 4.

Images