Data reconciliation method and device, electronic equipment and storage medium

Abstract

The application provides a data reconciliation method and device, electronic equipment and storage medium, and relates to the technical field of data processing. The method comprises the following steps: in response to detecting a target exchange task, reading a first offset value of a first message queue; in response to detecting that the target exchange task ends, reading a second offset value from the first message queue; and reconciling objects matched with the target exchange task in a source end and a destination end according to the first offset value and the second offset value to obtain a reconciliation result. Thus, the offset values of the first message queue before and after the storage object exchange (such as data exchange and file exchange) can be stored by a control node, the data reconciliation is performed according to the offset values of the first message queue before and after the object exchange, and no additional batch field needs to be stored in the destination end, so that the invasiveness to the destination end can be reduced.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data reconciliation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of big data technology, the need for data exchange between databases or between file servers is becoming increasingly urgent. ETL (Extract-Transform-Load, a data warehouse technology used to describe the process of extracting, transforming, and loading data from the source to the destination) tools has also been used more widely than ever before.

[0003] In data exchange scenarios, due to network isolation between different departments (such as departments in different provinces), data exchange is typically facilitated by a central database. The accuracy and consistency of this data exchange are essential for ensuring the smooth operation of subsequent business processes. Therefore, after data exchange, cross-gateway data reconciliation is required, which involves reconciling the synchronized business data between the source and destination databases across different gateways.

[0004] In related technologies, after the data exchange is completed, a batch of business data is extracted from the target database, and the extracted business data is reconciled with the corresponding batch of business data in the central database.

[0005] However, this reconciliation method is highly intrusive to the target database, requiring the storage of additional batch fields in the target database, which are often unrelated to the actual business. Summary of the Invention

[0006] The purpose of this application is to at least partially solve one of the aforementioned technical problems.

[0007] To this end, this application proposes a data reconciliation method, apparatus, electronic device, and storage medium to enable the central control node to store the offset values ​​of the first message queue before and after object exchange (such as data exchange or file exchange), and to perform data reconciliation based on the offset values ​​of the first message queue before and after object exchange, without storing additional batch fields in the destination end, thereby reducing the intrusion on the destination end.

[0008] The first aspect of this application proposes a data reconciliation method, including:

[0009] In response to the detection of a target exchange task, a first offset value of the first message queue is read; wherein the first offset value is used to indicate the storage location of the last written object in the first message queue; the target exchange task is used to exchange objects between the source and the destination through the first message queue;

[0010] In response to the detection that the target exchange task has ended, a second offset value is read from the first message queue; wherein the second offset value is used to indicate the storage location of the last object in the target exchange task in the first message queue;

[0011] Based on the first offset value and the second offset value, the objects in the source end and the destination end that match the target exchange task are reconciled to obtain the reconciliation result.

[0012] A second aspect of this application provides a data reconciliation device, comprising:

[0013] A first reading module is configured to read a first offset value of a first message queue in response to the detection of a target exchange task; wherein the first offset value is used to indicate the storage location of the last written object in the first message queue; the target exchange task is used to exchange objects between a source and a destination through the first message queue;

[0014] The second reading module is used to read a second offset value from the first message queue in response to detecting the end of the target exchange task; wherein the second offset value is used to indicate the storage location of the last object in the target exchange task in the first message queue;

[0015] The reconciliation module is used to reconcile the objects in the source end and the destination end that match the target exchange task based on the first offset value and the second offset value, so as to obtain the reconciliation result.

[0016] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the data reconciliation method as described in the first aspect.

[0017] The fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the data reconciliation method as described in the first aspect.

[0018] A fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the data reconciliation method described in the first aspect of this application.

[0019] The technical solutions provided by the embodiments of this application bring at least the following beneficial effects:

[0020] By reading the first offset value from the first message queue when a target exchange task is detected, and reading the second offset value from the first message queue when the target exchange task is detected to be completed, and then reconciling the objects matching the target exchange task in the source and destination ends based on the first and second offset values ​​to obtain the reconciliation results, the central control node can store the offset values ​​of the first message queue before and after data exchange, and perform data reconciliation based on the offset values ​​of the first message queue before and after object exchange (such as data exchange or file exchange), without storing additional batch fields in the destination end, thus reducing the intrusion on the destination end.

[0021] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0022] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0023] Figure 1 A flowchart illustrating a data reconciliation method provided in an embodiment of this application;

[0024] Figure 2 A flowchart illustrating another data reconciliation method provided in this application embodiment;

[0025] Figure 3 A flowchart illustrating another data reconciliation method provided in this application embodiment;

[0026] Figure 4 A flowchart illustrating another data reconciliation method provided in this application embodiment;

[0027] Figure 5 A flowchart illustrating another data reconciliation method provided in this application embodiment;

[0028] Figure 6 A flowchart illustrating another data reconciliation method provided in this application embodiment;

[0029] Figure 7 This is a schematic diagram of the architecture of the data reconciliation system provided in the embodiments of this application;

[0030] Figure 8 This is a schematic diagram of the business data flow provided in the embodiments of this application;

[0031] Figure 9 A schematic diagram illustrating the storage format of the documents provided in this application;

[0032] Figure 10 A flowchart illustrating another data reconciliation method provided in this application embodiment;

[0033] Figure 11 A flowchart illustrating another data reconciliation method provided in this application embodiment;

[0034] Figure 12 This is a schematic diagram of the structure of a data reconciliation device according to an embodiment of this application;

[0035] Figure 13 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0036] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0037] Currently, in data exchange scenarios, cross-gateway data reconciliation can be achieved through the following three methods:

[0038] The first method involves reconciling data after the data exchange is completed by comparing the amount of synchronized data with the corresponding batch data in the central database.

[0039] The second method involves extracting business data from the source and destination databases into the reconciliation database and then reconciling the data.

[0040] The third method involves extracting a batch of business data from the target database and reconciling the extracted business data with the corresponding batch of business data in the central database.

[0041] However, the first method only compares the amount of data to determine whether the business data exchanged between the source database and the destination database is consistent, but it cannot determine whether there is any missing or redundant business data in the destination database.

[0042] The second approach puts significant pressure on the reconciliation database's disk and performance when dealing with large volumes of data or reconciliation tasks. If a central database for the exchange tasks is used in the actual scenario, this could affect the execution of those tasks. Furthermore, in incremental exchange scenarios, since it's unclear which business data is involved in the current exchange task, it's necessary to extract all business data from the source database and the destination database for comparison. This puts pressure on the source database, destination database, and reconciliation database.

[0043] The third method is intrusive to the target database, and business departments are often unwilling to cooperate in adding such highly intrusive fields that are completely unrelated to business operations.

[0044] To address at least one of the aforementioned problems, this application proposes a data reconciliation method, apparatus, and electronic device. Before describing the embodiments of this application in detail, for ease of understanding, commonly used technical terms are first introduced:

[0045] A business timestamp refers to the timestamp carried in business data. The business timestamp can be the update time of the business data, the entry time (i.e., the time when it was stored in the database), or the occurrence time of the business event to which the business data belongs.

[0046] For example, when business data is updated, the business timestamp can be the update time of the business data (i.e., the latest or most recent update time), while when business data is not updated, the business timestamp can be the creation time or the time the business data is entered into the database.

[0047] A file timestamp can be either the file's update time or its creation time.

[0048] The following is combined Figure 1 This application provides a detailed explanation of the data reconciliation method provided.

[0049] Figure 1 This is a flowchart illustrating a data reconciliation method provided in an embodiment of this application.

[0050] The data reconciliation method of this application embodiment can be applied to the central control node or management node.

[0051] like Figure 1 As shown, this data reconciliation method includes the following steps:

[0052] Step S101: In response to detecting a target exchange task, read the first offset value of the first message queue; wherein the first offset value is used to indicate the storage location of the last written object in the first message queue.

[0053] In this embodiment, the target exchange task is used to exchange objects between the source and destination ends via a first message queue. Specifically, the target exchange task writes objects from the source end that match the target exchange task into the first message queue, and writes objects from the first message queue that match the target exchange task into the destination end.

[0054] The objects can include, but are not limited to, business data, files, etc.

[0055] In this embodiment of the application, the first message queue may be, for example, a Kafka (a high-throughput distributed publish-subscribe messaging system) message queue.

[0056] In this embodiment, the offset value of the first message queue (referred to as the first offset value in this application) is used to indicate the storage location of the last written object in the first message queue. For example, the first offset value is marked as offset1.

[0057] It should be noted that after each object is stored or written to the first message queue, the object has a corresponding offset value in the first message queue. This offset value is used to indicate the storage location of the object in the first message queue.

[0058] For example, taking objects as business data as an example, if the offset of business data 1 is 0, the offset of business data 2 is 1, and the offset of business data 3 is 2, then it means that business data 1 is the first piece of business data written in the first message queue, business data 2 is the second piece of business data written in the first message queue, and business data 3 is the third piece of business data written in the first message queue.

[0059] In the embodiments of this application, when an update is detected in an object (such as business data or file) in the source end, a target exchange task can be generated. The target exchange task is used to indicate the updated object in the source end. Alternatively, the target exchange task can be manually triggered. For example, the target exchange task can carry a query period. The target exchange task is used to indicate objects in the source end whose timestamps (such as business timestamps or file timestamps) are within the query period. Alternatively, the target exchange task can also be used to indicate all objects in the source end, etc. This application does not limit the generation method of the target exchange task.

[0060] In this embodiment of the application, when a target exchange task is detected, the central control node can send the target exchange task to the exchange node corresponding to the source end (referred to as the first exchange node in this application). Correspondingly, after receiving the target exchange task, the first exchange node can respond to the target exchange task by obtaining the object matching the target exchange task from the source end, and sorting the object matching the target exchange task according to the timestamp of the object matching the target exchange task (such as business timestamp, file timestamp). For example, sorting the objects according to the timestamp value from smallest to largest, and writing the sorted objects sequentially into the first message queue.

[0061] In this embodiment of the application, the central control node can also send a target exchange task to the second exchange node corresponding to the destination. Accordingly, after receiving the target exchange task, the second exchange node can respond to the target exchange task by reading the object matching the target exchange task from the first message queue and writing it to the destination. Thus, the exchange of objects between the source and destination can be realized through the first message queue.

[0062] In this embodiment of the application, when a target exchange task is detected, the central control node can also obtain or read the first offset value of the first message queue; wherein, the first offset value is used to indicate the storage location of the last written object in the first message queue.

[0063] Step S102: In response to detecting the end of the target exchange task, read the second offset value from the first message queue; wherein the second offset value is used to indicate the storage location of the last object in the target exchange task in the first message queue.

[0064] In this embodiment, when the central control node detects the end of the target exchange task, it can read a second offset value from the first message queue. The second offset value indicates the storage location of the last object in the target exchange task within the first message queue. Specifically, the second offset value indicates the storage location of the last object written to the first message queue among the objects matching the target exchange task. For example, the second offset value can be marked as offset2.

[0065] Step S103: Based on the first offset value and the second offset value, reconcile the objects in the source and destination ends that match the target exchange task to obtain the reconciliation result.

[0066] In this embodiment of the application, the objects matching the target exchange task in the source and destination ends can be reconciled based on the first offset value and the second offset value to obtain the reconciliation result.

[0067] The data reconciliation method of this application embodiment reads a first offset value from a first message queue in response to the detection of a target exchange task, and reads a second offset value from the first message queue when the target exchange task is detected to be finished. Based on the first and second offset values, reconciliation is performed on objects matching the target exchange task in the source and destination ends to obtain a reconciliation result. Therefore, the central control node can store the offset values ​​of the first message queue before and after object exchange (such as data exchange or file exchange), and perform data reconciliation based on these offset values, without needing to store additional batch fields in the destination end, thus reducing intrusion on the destination end.

[0068] In one possible implementation of the embodiments of this application, when the object is business data, the source end can be a source database and the destination end can be a destination database. In order to clearly explain how the above embodiments of this application reconcile the business data in the source database and the destination database that match the target exchange task based on the first offset value and the second offset value, this application also proposes a data reconciliation method.

[0069] Figure 2 This is a flowchart illustrating another data reconciliation method provided in an embodiment of this application.

[0070] like Figure 2 As shown, this data reconciliation method may include the following steps:

[0071] Step S201: In response to the detection of a target exchange task, read the first offset value of the first message queue.

[0072] The first offset value is used to indicate the storage location of the last written business data in the first message queue, and the target exchange task is used to exchange data between the source database and the destination database through the first message queue.

[0073] In step S202, in response to the detection that the target exchange task has ended, the second offset value is read from the first message queue.

[0074] The second offset value is used to indicate the storage location of the last business data in the target exchange task in the first message queue.

[0075] The explanation of steps S201 to S202 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0076] In step S203, in response to the target exchange task being a full exchange, the first business data stream is extracted from the first message queue based on the first offset value and the second offset value.

[0077] In this embodiment of the application, the task type of the target exchange task may include full exchange (i.e., business data matching the target exchange task, which is all business data in the source database) and incremental exchange (i.e., business data matching the target exchange task, which is part of the business data in the source database, such as newly added business data in the source database).

[0078] In this context, the offset of each piece of first business data in the first business data stream is located between the first offset (offset1) and the second offset (offset2) in the first message queue. For example, offset1+1 is the offset of the first piece of business data in the target exchange task, and business data with offsets of offset1+1, offset1+2, offset1+3, ..., offset2-1, offset2 can be extracted.

[0079] In this embodiment of the application, when the task type of the target exchange task is full exchange, the first message queue includes all business data in the source database. Therefore, the first business data stream can be extracted from the first message queue. Specifically, the first business data stream can be extracted from the first message queue according to the first offset value and the second offset value. The offset value of each piece of first business data in the first business data stream in the first message queue is between the first offset value and the second offset value. For example, the offset value of each piece of first business data in the first business data stream in the first message queue is within [first offset value + 1, second offset value].

[0080] The offset value is used to indicate the storage location of the first business data in the first message queue.

[0081] Step S204: Extract the second business data stream from the destination database, wherein the second business data stream includes all the second business data in the destination database.

[0082] In this embodiment, a second business data stream can be extracted from the target database. The second business data stream includes all business data in the target database (referred to as the second business data in this application). That is, all business data can be extracted from the target database to form the second business data stream.

[0083] Step S205: Perform data reconciliation on the first business data stream and the second business data stream to obtain the reconciliation result.

[0084] In this embodiment of the application, data reconciliation can be performed on the first business data stream and the second business data stream to obtain the reconciliation result.

[0085] It should be noted that in the scenario of full data exchange, both the first business data stream and the second business data stream contain a large amount of business data. If a large amount of business data is compared at the same time, the processing burden on the device is large. Therefore, in one possible implementation of the embodiments of this application, in order to reduce the processing burden and processing pressure on the device, a sliding window can be used to slide and capture the first business data stream and the second business data stream, and data reconciliation is performed on the business data within one sliding window at a time.

[0086] As an example, whenever a set period is reached (e.g., every ten seconds), a first sliding window of a first set time length can be used to capture the first business data stream and the second business data stream. The first sliding window slides for a second set time length each time. The captured first business data and second business data within the first sliding window are reconciled to obtain the reconciliation result.

[0087] There are no restrictions on the relationship between the first set time length and the second set time length. For example, the first set time length can be equal to the second set time length, or the first set time length can be less than the second set time length, or the first set time length can be greater than the second set time length.

[0088] For example, taking a first set time length equal to a second set time length, both 10 seconds, the first and second business data within the time period [a, a+10] can be reconciled for the first time. The second time, the first and second business data within the time period [a+10, a+20] can be reconciled for the second time. The third time, the first and second business data within the time period [a+20, a+30] can be reconciled for the third time. And so on.

[0089] Where 'a' is the initial timestamp.

[0090] It's important to note that periodically sliding the first sliding window avoids situations where one data stream arrives before the other, leading to inaccurate reconciliation results. For example, sliding the first window every 10 seconds allows for a 10-second wait before reconciliation, improving accuracy.

[0091] The data reconciliation method in this application embodiment can reconcile business data in a full data exchange scenario, thus meeting actual data reconciliation needs.

[0092] To clearly illustrate how the first business data stream and the second business data stream are reconciled in the above embodiments of this application, this application also proposes a data reconciliation method.

[0093] Figure 3 This is a flowchart illustrating another data reconciliation method provided in an embodiment of this application.

[0094] like Figure 3 As shown, this data reconciliation method may include the following steps:

[0095] Step S301: In response to the detection of a target exchange task, read the first offset value of the first message queue.

[0096] The first offset value is used to indicate the storage location of the last piece of business data written in the first message queue, and the target exchange task is used to exchange data between the source database and the destination database through the first message queue.

[0097] In step S302, in response to the detection that the target exchange task has ended, the second offset value is read from the first message queue.

[0098] The second offset value is used to indicate the storage location of the last business data in the target exchange task in the first message queue.

[0099] In step S303, in response to the target exchange task being a full exchange, the first business data stream is extracted from the first message queue based on the first offset value and the second offset value.

[0100] In this context, the offset value of each first business data in the first business data stream within the first message queue is between the first offset value and the second offset value.

[0101] Step S304: Extract the second business data stream from the destination database, wherein the second business data stream includes all the second business data in the destination database.

[0102] Step S305: Whenever a set period is reached, a first sliding window of a first set time length is used to capture the first business data stream and the second business data stream, wherein the first sliding window slides for a second set time length.

[0103] The explanation of steps S301 to S305 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0104] Step S306: Establish a relationship between the first business data and the second business data that are captured and match the data identifiers within the first sliding window.

[0105] Among them, the data identifier is used to uniquely identify business data. For example, the data identifier can be the primary key, unique key, ID, etc. of business data.

[0106] In this embodiment, the data identifier of the first service data in the first sliding window can be matched with the data identifier of the second service data, and an association relationship can be established between the first service data and the second service data whose data identifiers match in the first sliding window.

[0107] Step S307: Compare the business fields of the first business data and the second business data that have a relationship, and compare the business timestamps of the first business data and the second business data that have a relationship.

[0108] In this embodiment of the application, the business fields of the first business data and the second business data that are related can be compared, and it can be determined whether the business fields of the first business data and the second business data that are related match. If the business fields of the first business data and the second business data that are related match, it indicates that the business fields of the second business data that are related synchronized to the destination database are not wrong. If the business fields of the first business data and the second business data that are related do not match, it indicates that the business fields of the second business data that are related synchronized to the destination database are wrong. At this time, step S308 can be executed.

[0109] In this embodiment of the application, the business timestamps of the first business data and the second business data that are related can also be compared, and it can be determined whether the business timestamps of the first business data and the second business data that are related match. If the business timestamps of the first business data and the second business data that are related match, it indicates that the business timestamp of the second business data that is related synchronized to the destination database is not wrong. If the business timestamps of the first business data and the second business data that are related do not match, it indicates that the business timestamp of the second business data that is related synchronized to the destination database is wrong. At this time, step S308 can be executed.

[0110] In step S308, if the business fields of the first business data and the second business data that are related do not match, and / or the business timestamps of the first business data and the second business data that are related do not match, a first reconciliation result is generated based on the first business data and the second business data that are related.

[0111] In this embodiment, if the business fields of the related first business data and the related second business data do not match, and / or the business timestamps of the related first business data and the related second business data do not match, it indicates that the related second business data synchronized to the destination database is incorrect. In this case, a first reconciliation result can be generated based on the related first business data and the related second business data. The first reconciliation result indicates that the related second business data in the destination database does not match the related first business data in the source database.

[0112] In one possible implementation of this application embodiment, when there is first business data in the first sliding window that has not been associated, it indicates that the first business data has been missed in synchronizing to the destination database. At this time, a second reconciliation result can be generated based on the first business data that has not been associated. The second reconciliation result is used to indicate that the destination database is missing the aforementioned first business data that has not been associated.

[0113] In one possible implementation of this application embodiment, when there is second business data in the first sliding window that has not been associated, it indicates that the second business data is redundant business data in the destination database, that is, the second business data does not exist in the source database. At this time, a third reconciliation result can be generated based on the second business data that has not been associated. The third reconciliation result is used to indicate that the second business data that has not been associated exists in the destination database.

[0114] Therefore, it can not only reconcile erroneous business data in the target database, but also missing and extra business data in the target database, thus improving the comprehensiveness and completeness of data reconciliation.

[0115] In one possible implementation of this application embodiment, at least one of the following can also be performed based on the reconciliation results:

[0116] The first item is to update the related second business data in the target database based on the first reconciliation result and the related first business data.

[0117] The second step involves writing the first business data, which has not yet been linked, into the target database based on the second reconciliation result.

[0118] Thirdly, based on the third reconciliation result, delete the second business data in the target database that has not been linked.

[0119] This can improve the consistency and integrity of business data synchronized or exchanged between the source and target databases.

[0120] It should be noted that during the process of writing business data from the source database into the first message queue, data loss or errors may occur. In this case, even if it is determined that the reconciliation is error-free based on the first and second business data streams, it does not indicate that the business data synchronized or exchanged between the source and target databases is completely consistent. Therefore, in one possible implementation of this application embodiment, in order to improve the accuracy of the reconciliation results and the consistency and integrity of the business data synchronized or exchanged between the source and target databases, when the task type of the target exchange task is a full exchange, a third business data stream can also be extracted from the source database. The third business data stream includes all the third business data in the source database, and data reconciliation is performed between the first and third business data streams to obtain the reconciliation results.

[0121] It should be noted that the data reconciliation method between the first business data stream and the third business data stream is similar to that between the first business data stream and the second business data stream, and their implementation principles are similar, so they will not be elaborated here.

[0122] The data reconciliation method of this application embodiment can realize the reconciliation of erroneous business data in the destination database, thus meeting the actual data reconciliation needs.

[0123] In one possible implementation of the embodiments of this application, when the object is business data, the source end can be a source database and the destination end can be a destination database. In order to clearly explain how the above embodiments of this application perform data reconciliation on the business data in the source database and the destination database that match the target exchange task based on the first offset value and the second offset value, this application also proposes a data reconciliation method.

[0124] Figure 4 This is a flowchart illustrating another data reconciliation method provided in an embodiment of this application.

[0125] like Figure 4 As shown, this data reconciliation method may include the following steps:

[0126] Step S401: In response to the detection of a target exchange task, read the first offset value of the first message queue.

[0127] The first offset value is used to indicate the storage location of the last written business data in the first message queue; the target exchange task is used to exchange data between the source database and the destination database through the first message queue.

[0128] In step S402, in response to the detection that the target exchange task has ended, the second offset value is read from the first message queue.

[0129] The second offset value is used to indicate the storage location of the last business data in the target exchange task in the first message queue.

[0130] Step S403: In response to the target exchange task being incremental exchange, the first business timestamp of the business data matching the first offset value is queried from the first message queue according to the first offset value.

[0131] In this embodiment, when the target exchange task is incremental exchange, the first business timestamp of the business data matching the first offset value can be queried from the first message queue based on the first offset value. For example, the first offset value is marked as offset1, offset1+1 is the offset value of the first business data in the target exchange task, and the first business timestamp can be the business timestamp of the first business data. That is, the offset value of the business data matching the first offset value is offset1+1, and the first business timestamp can be recorded as (offset1+1)timestamp. Therefore, the business data stream to be reconciled can be extracted subsequently based on the first business timestamp.

[0132] The offset value indicates the storage location of the business data in the first message queue.

[0133] Step S404: Based on the second offset value, query the second business timestamp of the business data that matches the second offset value from the first message queue.

[0134] Similarly, the second business timestamp of the business data matching the second offset value can be retrieved from the first message queue based on the second offset value. For example, if the second offset value is marked as offset2, where offset2 is the offset value of the last business data in the target exchange task, the second business timestamp can be the business timestamp of that last business data. That is, if the offset value of the business data matching the second offset value is offset2, the second business timestamp can be recorded as offset2 timestamp.

[0135] Step S405: Extract the fourth business data stream from the source database and the fifth business data stream from the destination database based on the first business timestamp and the second business timestamp.

[0136] It should be noted that when the target exchange task is an incremental exchange, the source database may generate multiple business data entries at the same time (e.g., adding N new business data entries). These multiple business data entries may be exchanged to the destination database through multiple exchange tasks. For example, the N business data entries generated at the same time may be exchanged to the destination database through two exchange tasks (e.g., exchange task 1 and exchange task 2). For instance, each exchange task is used to exchange N / 2 business data entries to the destination database. For exchange task 1, when exchange task 1 ends, N / 2 business data entries matching exchange task 1 can be extracted from the first message queue based on the first and second offset values ​​corresponding to exchange task 1. Then, based on the business timestamps of the extracted N / 2 business data entries, N / 2 business data entries are extracted from the destination database. For exchange task 2, when exchange task 2 ends, N / 2 business data entries matching exchange task 2 can be extracted from the first message queue based on the first and second offset values ​​corresponding to exchange task 2. However, based on the extracted N / 2 business data entries... The business timestamp of the business data can be used to extract N business data from the destination database (N / 2 business data exchanged by exchange task 1 + N / 2 business data exchanged by exchange task 2). That is, when the first exchange task is executed, only N / 2 business data at the same time are imported into the destination database. However, after the two exchange tasks are executed, N business data at the same time are imported into the destination database. At this time, for exchange task 2, the business data extracted from the first message queue will be reconciled with the business data extracted from the destination database, which will lead to reconciliation errors. That is, it will be mistakenly believed that the business data exchanged by exchange task 1 to the destination database is redundant business data.

[0137] In response to the above situation, in order to improve the comprehensiveness, completeness and accuracy of the reconciliation results, the fourth business data stream can be extracted from the source database based on the first business timestamp and the second business timestamp. The business timestamp of each piece of fourth business data in the fourth business data stream is between the first business timestamp and the second business timestamp, that is, the business timestamp of each piece of fourth business data in the fourth business data stream is within [first business timestamp, second business timestamp].

[0138] Similarly, the fifth business data stream can be extracted from the target database based on the first business timestamp and the second business timestamp. The business timestamp of each piece of fifth business data in the fifth business data stream is between the first business timestamp and the second business timestamp, that is, the business timestamp of each piece of fifth business data in the fifth business data stream is within [first business timestamp, second business timestamp].

[0139] Step S406: Perform data reconciliation between the fourth business data stream and the fifth business data stream to obtain the reconciliation results.

[0140] In this embodiment, data reconciliation can be performed on the fourth and fifth business data streams to obtain reconciliation results. It should be noted that the data reconciliation method for the fourth and fifth business data streams is similar to that for the first and second business data streams, and their implementation principles are similar, so it will not be elaborated upon here.

[0141] The data reconciliation method of this application embodiment can not only reconcile business data in a full exchange scenario, but also reconcile business data in an incremental exchange scenario, thereby improving the flexibility and applicability of the method.

[0142] In one possible implementation of this application embodiment, when the object is business data, the source end can be a source database and the destination end can be a destination database. In order to clearly explain how the business data in the source database and the destination database that matches the target exchange task are reconciled in this application embodiment to obtain the reconciliation result, this application also proposes a reconciliation method.

[0143] Figure 5 This is a flowchart illustrating another reconciliation method provided in an embodiment of this application.

[0144] like Figure 5 As shown, this reconciliation method may include the following steps:

[0145] Step S501: In response to the detection of a target exchange task, read the first offset value of the first message queue.

[0146] The first offset value is used to indicate the storage location of the last written business data in the first message queue; the target exchange task is used to exchange data between the source database and the destination database through the first message queue.

[0147] The explanation of step S501 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0148] In step S502, in response to the fact that the target exchange task has not ended, the sixth service data stream is extracted from the first message queue according to the first offset value; wherein, the offset value corresponding to each sixth service data in the sixth service data stream is greater than the first offset value.

[0149] It should be noted that for real-time reconciliation scenarios, when there is always incremental business data in the source database, the target exchange task may not stop. In this case, it is necessary to extract the business data to be reconciled from the first message queue whose offset value is greater than the first offset value.

[0150] In this application, before the target exchange task is completed, the sixth service data stream can be extracted from the first message queue according to the first offset value; wherein, the offset value corresponding to each sixth service data in the sixth service data stream is greater than the first offset value (offset1), that is, service data with offset values ​​of offset1+1, offset1+2, offset1+3, ... in the first message queue can be extracted.

[0151] Step S503: Extract the seventh service data stream from the second message queue according to the third offset value; wherein, the third offset value is used to indicate the storage location of the last written service data in the second message queue before the target exchange task is started, and each seventh service data in the seventh service data stream is the updated service data obtained from the target database in response to the detection of the target exchange task.

[0152] In this embodiment, a second message queue can be used to record business data written to the destination database. For example, before the target exchange task starts, a third offset value of the second exchange task can be read, where the third offset value indicates the storage location of the last piece of business data written in the second message queue. Furthermore, when the target exchange task is detected, newly added business data in the destination database can be sensed, and this newly added business data can be written to the second message queue. In other words, the second message queue contains updated business data from the destination database.

[0153] In this embodiment, a seventh service data stream can be extracted from the second message queue based on a third offset value. Each seventh service data item in the seventh service data stream is updated service data obtained from the target database in response to the detection of a target exchange task. For example, if the third offset value is set to offset3, service data with offset values ​​of offset3+1, offset3+2, offset3+3, ... can be extracted from the second message queue.

[0154] Step S504: Perform data reconciliation between the sixth business data stream and the seventh business data stream to obtain the reconciliation results.

[0155] In this embodiment of the application, data reconciliation can be performed on the sixth business data stream and the seventh business data stream to obtain the reconciliation results.

[0156] It should be noted that the data reconciliation methods for the sixth and seventh business data streams are similar to those for the first and second business data streams, and their implementation principles are similar, so they will not be elaborated here.

[0157] The reconciliation method in this application can not only reconcile business data in full exchange scenarios and incremental exchange scenarios, but also reconcile business data in real-time reconciliation scenarios, which can further improve the flexibility and applicability of the method.

[0158] In any embodiment of this application, for real-time reconciliation scenarios, the business data in business database A and business database B contain primary keys or unique keys and timestamp fields.

[0159] In this real-time reconciliation scenario, if incremental business data consistently exists in the source database, the exchange task may not stop. Therefore, it's necessary to monitor changes in the business data in real time. Before starting the exchange task, record the starting position offset1 in the first message queue (e.g., message queue 1). If real-time reconciliation is enabled, prioritize starting the incremental task from the destination database to the second message queue (e.g., message queue 2) to write the incremental business data from the destination database to message queue 2. During data reconciliation, business data stream 2 can be extracted from message queue 2.

[0160] After ensuring the incremental task for incremental business data -> message queue 2 in the destination database is started, start the incremental task for incremental data -> message queue 1 in the source database. During data reconciliation, business data stream 1 can be obtained by consuming data starting from offset1.

[0161] By comparing business data flow 1 and business data flow 2, and using state programming, it is possible to detect lost or delayed business data in the destination database in real time and issue alerts to users.

[0162] To clearly illustrate any of the above embodiments of this application, this application also proposes a data reconciliation method.

[0163] Figure 6 This is a flowchart illustrating another data reconciliation method provided in an embodiment of this application.

[0164] like Figure 6 As shown, when a target exchange task is detected, this data reconciliation method may further include the following steps:

[0165] Step S601: In response to detecting the target exchange task, send the first exchange task to the first exchange node corresponding to the source end.

[0166] The first exchange task is used by the first exchange node to obtain objects that match the first exchange task from the source end, sort the objects that match the first exchange task according to the timestamp of the objects that match the first exchange task, and write the sorted objects sequentially into the first message queue.

[0167] In this embodiment, when the central control node detects an update to an object in the source database, it can generate a first exchange task. The first exchange task is used to indicate the updated object in the source database. Alternatively, the first exchange task can be manually triggered. For example, the first exchange task can carry a query period and is used to indicate objects in the source database whose timestamps are within the query period. Or, the first exchange task can also be used to indicate all objects in the source database, etc. This application does not limit the generation method of the first exchange task.

[0168] In this embodiment of the application, the central control node can send a first exchange task to the first exchange node corresponding to the source end. Correspondingly, after receiving the first exchange task, the first exchange node can respond to the first exchange task by obtaining the object matching the first exchange task from the source end, and sorting the object matching the first exchange task according to the timestamp of the object matching the first exchange task. For example, sorting the objects according to the timestamp value from smallest to largest, and writing the sorted objects sequentially into the first message queue.

[0169] As one possible implementation, taking objects as business data as an example, the source can be a source database and the destination can be a destination database. The central control node can send a first exchange task to the first exchange node corresponding to the source database. Correspondingly, after receiving the first exchange task, the first exchange node can respond to the first exchange task by retrieving the business data matching the first exchange task from the source database, and sorting the business data matching the first exchange task according to the business timestamp of the business data matching the first exchange task. For example, the business data can be sorted from smallest to largest according to the value of the business timestamp, and the sorted business data can be written sequentially into the first message queue.

[0170] Step S602: Send a second exchange task to the second exchange node corresponding to the destination. The second exchange task is used by the second exchange node to read the object that matches the first exchange task from the first message queue and write it into the destination database.

[0171] As an example, taking objects as business data, the source can be a source database, and the destination can be a destination database. The central control node can also send a second exchange task to the second exchange node corresponding to the destination database. Correspondingly, upon receiving the second exchange task, the second exchange node can respond by reading the business data matching the second exchange task from the first message queue and writing it to the destination database. The business data matching the second exchange task is the same as the business data matching the first exchange task.

[0172] In this embodiment of the application, when the end of the target exchange task is detected, the second offset value is read from the first message queue. Specifically, the second offset value can be read from the first message queue when the end of the first exchange task and / or the second exchange task is detected.

[0173] In this embodiment of the application, when the central control node detects that the first exchange task and / or the second exchange task has ended, it can read the second offset value from the first message queue, wherein the second offset value is used to indicate the storage location of the last written business data in the first message queue.

[0174] The data reconciliation method in this application embodiment can achieve effective exchange of business data by having the central control node issue exchange tasks to different exchange nodes.

[0175] In any embodiment of this application, it is possible to complete full or incremental data reconciliation without occupying additional reconciliation database resources and without adding batch fields to the target database, and accurately identify lost business data.

[0176] Specifically, Kafka (message queue) can be used instead of a central database as a relay for cross-gateway data exchange. The central control node records the topic offset 1 value before the source database -> Kafka exchange synchronization starts and the topic offset 2 value after the exchange synchronization is completed. The reconciliation node extracts the business data between offset 2 and offset 1 from Kafka and compares it with the corresponding business data in the destination database, and uses stream computing to obtain the reconciliation result.

[0177] As an example, the structure of a data reconciliation system can be as follows: Figure 7 As shown, the source can be a source database, and the destination can be a destination database. Figure 7 Taking the source database as business database A, the first exchange node as exchange node A, the destination database as business database B, and the second exchange node as exchange node B as an example, the business data in business database A and business database B contain primary keys or unique keys and timestamp fields.

[0178] 1. Central control node (i.e. Figure 7 The management node in the system issues an exchange task from business database A to Kafka to exchange node A, and an exchange task from Kafka to business database B to exchange node B. The exchange task from business database A to Kafka requires sorting each piece of business data according to its timestamp field, and then writing the sorted data to the Kafka message queue to ensure that the business data is written to the Kafka message queue in ascending order of timestamp.

[0179] 2. Based on the characteristics of the Kafka message queue, each piece of business data written has a corresponding offset, and the offset value increases sequentially as data is written. Therefore, before starting the exchange task, the central control node can read the current offset1 (index value) of the corresponding topic from the Kafka message queue.

[0180] 3. After the exchange task between gateway A and gateway B is completed, the central control node reads offset2 (index value) from the Kafka message queue.

[0181] 4. Once the central control node detects that the exchange task has been completed, it sends a reconciliation task to the reconciliation service (i.e., the reconciliation node).

[0182] In a full exchange scenario, the data between offset2 and offset1 in the Kafka message queue is all the business data in this exchange. Therefore, the business data between offset2 and offset1 in the Kafka message queue can be directly used as business data stream 1 to extract all the business data from the business database B, forming business data stream 2, and then business data stream 1 and business data stream 2 can be compared.

[0183] In the incremental exchange scenario, the reconciliation service determines the batch range of this exchange through the timestamp fields corresponding to offset2 and offset1. It can query business data within the interval [(offset1+1)timestamp, offset2timestamp] from business database A to form business data stream 1. It also queries business data within the interval [(offset1+1)timestamp, offset2timestamp] from business database B to form business data stream 2. Then, business data stream 1 and business data stream 2 are compared.

[0184] 5. Associate business data stream 1 and business data stream 2 using a primary key or unique key, and use a timestamp field as a watermark semantic. Use stateful programming (ValueState) to store the state (e.g., "arrived") of the first arriving business data and establish a timer (e.g., timer duration of 10 seconds). If the other business data stream arrives within the time limit, it will output and the timer will be deleted; otherwise, the timer will be triggered to output to the side output stream. In the timer, all business data that has not yet arrived will be output to the side output stream, thus achieving a full outer join. If A has arrived but B has not, output will be performed; if A has not arrived but B has arrived, no output will be performed, thus achieving a left join. Conversely, this achieves a right join effect.

[0185] 6. By using the left join output stream, we can determine the missing or changed business data in business database B. By using the right join output stream, we can determine the redundant business data in business database B.

[0186] Take business data stream 1 (stream1) and business data stream 2 (stream2) as an example Figure 8 As shown in the example, reconciling the data between stream1 and stream2 yields the following results: (2, null) (meaning the data with primary key 2 in stream1 has arrived, but not in stream2), (null, 3) (meaning the data with primary key 3 in stream2 has arrived, but not in stream1), (5, null) (meaning the data with primary key 5 in stream1 has arrived, but not in stream2), and (null, 2) (meaning the data with primary key 2 in stream2 has arrived, but not in stream1).

[0187] Since the business data with primary key 2 in stream2 is actually changing data, that is, the timestamp of the business data with primary key 2 in stream2 (at time t2) is inconsistent with the timestamp of the business data with primary key 2 in stream1 (at time t1), it can still be considered that the business database B is missing the business data with primary key 2 at time t1 and has the business data with primary key 2 at time t2.

[0188] Based on the above reconciliation results, the data reconciliation system can choose to perform a correction operation, deleting business data in business database B with primary or unique keys 2 and 3, and adding business data in stream1 with primary or unique keys 2 and 5, thus ensuring that the data in business database A and business database B are consistent.

[0189] In summary, reconciliation results can be obtained by extracting business data between topic offsets and performing streaming calculations. This does not require additional reconciliation database resources or adding batch fields to the target database. It can complete full or incremental data reconciliation and accurately identify lost business data.

[0190] In one possible implementation of this application's embodiments, when the object is a file, the source end can be a source file server, the destination end can be a destination file server, and the file can be written to a first message queue through multiple messages. For example, the file can be split into multiple binary arrays, and each binary array can be a message. For example, the file can be as follows: Figure 9 The format shown is written to the first message queue, where, Figure 9 Each cell in the table represents a message. To illustrate how this application reconciles files matching the target exchange task in the source and destination file servers based on the first and second offset values, this application also proposes a data reconciliation method.

[0191] Figure 10This is a flowchart illustrating another data reconciliation method provided in an embodiment of this application.

[0192] like Figure 10 As shown, this data reconciliation method may include the following steps:

[0193] Step S1001: In response to the detection of a target exchange task, read the first offset value of the first message queue.

[0194] The first offset value is used to indicate the storage location of the last file currently written in the first message queue. It can be understood that since files are written to the first message queue in multiple messages, the first offset value can be used to indicate the storage location of the last message in the last file currently written to in the first message queue.

[0195] The target exchange task is used to exchange files between the source file server and the destination file server through the first message queue.

[0196] In step S1002, in response to the detection that the target exchange task has ended, the second offset value is read from the first message queue.

[0197] The second offset value is used to indicate the storage location of the last file in the target exchange task within the first message queue. It can be understood that since the file is written to the first message queue via multiple messages, the second offset value can be used to indicate the storage location of the last message in the last file within the first message queue.

[0198] The explanation of steps S1001 to S1002 can be found in the relevant description in any embodiment of this application, and will not be repeated here.

[0199] Step S1003: Extract a first message stream from the first message queue according to the first offset value and the second offset value; wherein the offset value corresponding to each message in the first message stream is between the first offset value and the second offset value.

[0200] In this embodiment of the application, a first message stream can be extracted from a first message queue based on a first offset value and a second offset value; wherein the offset value corresponding to each message in the first message stream is between the first offset value and the second offset value. For example, the offset value of each message in the first message stream in the first message queue is within [first offset value + 1, second offset value].

[0201] Step S1004: Generate a second message stream based on the file written to the destination file server.

[0202] In this embodiment of the application, a second message stream can be generated based on the file written in the destination file server.

[0203] In one possible implementation of this application, the second message stream is generated as follows: All files (referred to as target files in this application) are extracted from the destination file server, and a hash operation is performed on each target file to obtain a hash value (referred to as a second hash value in this application). For any target file, a second message can be generated based on the filename, second hash value, and file timestamp corresponding to that target file. Furthermore, the second messages can be sorted according to their corresponding file timestamps to obtain the second message stream. For example, the second messages can be sorted from smallest to largest based on the file timestamp values ​​to obtain the second message stream.

[0204] Therefore, it is possible to reconcile the message stream based on the hash value and file timestamp of each message in the message stream, without having to reconcile the message content in the message stream, which can improve reconciliation efficiency.

[0205] In another possible implementation of this application, the second message stream is generated as follows: The central control node can read and store the fourth offset value of the third message queue before the target exchange task starts. The fourth offset value indicates the storage location of the last message written in the third message queue before the target exchange task starts. Furthermore, the central control node can read and store the fifth offset value of the third message queue when the target exchange task ends or after it ends. The fifth offset value indicates the storage location of the last message written in the third message queue when the target exchange task ends or after it ends. Therefore, in this application, an initial message stream can be extracted from the third message queue based on the fourth and fifth offset values. Each message in the initial message stream is generated based on a file written to the destination file server. That is, in this application, when a target exchange task is detected, a file can be extracted from the destination file server and written to the third message queue. When the target exchange task is detected to be finished or after it ends, an initial message stream can be extracted from the third message queue. The offset value of each message in the initial message stream is between [fourth offset value + 1, fifth offset value].

[0206] In this application, multiple messages belonging to the same file in the initial message stream can be merged, hashed to obtain a second hash value for the same file, and then a second message can be generated based on the filename, the second hash value, and the file timestamp. This allows the second messages to be sorted according to their corresponding file timestamps, resulting in a second message stream. For example, the second messages can be sorted from smallest to largest based on their file timestamp values ​​to obtain the second message stream.

[0207] Therefore, a second message stream can be generated in different ways, which can improve the flexibility and applicability of the method.

[0208] Step S1005: Reconcile the first message stream and the second message stream to obtain the reconciliation result.

[0209] In this embodiment of the application, the first message stream and the second message stream can be reconciled to obtain the reconciliation result.

[0210] The data reconciliation method of this application embodiment can not only reconcile business data in full exchange scenarios, incremental exchange scenarios, and real-time reconciliation scenarios, but also reconcile files in file exchange scenarios, which can further improve the flexibility and applicability of the method.

[0211] To clearly illustrate how the first message stream and the second message stream are reconciled in the embodiments of this application to obtain the reconciliation result, this application also proposes a reconciliation method.

[0212] Figure 11 This is a flowchart illustrating another reconciliation method provided in an embodiment of this application.

[0213] like Figure 11 As shown, step S1005 may include the following steps:

[0214] Step S1101: After merging multiple messages belonging to the same file in the first message stream, perform a hash operation to obtain the first hash value of the same file.

[0215] In this embodiment, multiple messages belonging to the same file in the first message stream can be merged to obtain a merged message corresponding to the same file. A hash operation can then be performed on the merged message to obtain the first hash value of the same file. For example, a hash algorithm (such as MD5 (Message-Digest Algorithm 5)) can be used to perform a hash operation on the merged message to obtain the first hash value of the same file.

[0216] Step S1102: Generate the first message based on the filename, first hash value and file timestamp corresponding to the same file.

[0217] In this embodiment of the application, a first message can be generated based on the file name, first hash value and file timestamp corresponding to the same file.

[0218] Step S1103: Sort each first message according to the file timestamp corresponding to each first message to obtain the updated first message stream.

[0219] In this embodiment of the application, the first messages can be sorted according to the file timestamps corresponding to each first message to obtain an updated first message stream. For example, the first messages can be sorted from smallest to largest according to the file timestamp values ​​to obtain the first message stream.

[0220] Step S1104: Reconcile the updated first message stream and second message stream to obtain the reconciliation result.

[0221] In this embodiment of the application, the updated first message stream and second message stream can be reconciled to obtain the reconciliation result.

[0222] It should be noted that in file exchange scenarios, both the first and second message streams contain a large number of messages. If a large number of messages are compared at the same time, the processing burden on the device will be large. Therefore, in one possible implementation of this application embodiment, in order to reduce the processing burden and processing pressure on the device, a sliding window can be used to slide and capture the updated first and second message streams, and only the messages in one sliding window are reconciled at a time.

[0223] As an example, whenever a set period is reached (e.g., every ten seconds), a second sliding window of a third set time length is used to capture the updated first and second message streams; wherein, the second sliding window slides for a fourth set time length each time; a relationship is established between the first and second messages with matching filenames within the second sliding window; the first hash value of the associated first message and the second hash value of the associated second message are compared, and the file timestamps of the associated first message and the second message are compared.

[0224] There are no restrictions on the relationship between the third and fourth set time lengths. For example, the third set time length can be equal to the fourth set time length, or the third set time length can be less than the fourth set time length, or the third set time length can be greater than the fourth set time length.

[0225] If the first hash value of the first message and the second hash value of the second message with a relationship do not match (indicating that the content of the file corresponding to the filename in the second message with a relationship in the destination file server is incorrect), and / or the file timestamp of the first message with a relationship does not match the file timestamp of the second message with a relationship (indicating that the timestamp of the file corresponding to the filename in the second message with a relationship in the destination file server is incorrect), a fourth reconciliation result can be generated based on the first message and the second message with a relationship; wherein, the fourth reconciliation result is used to indicate that the file corresponding to the filename in the second message with a relationship in the destination file server and the file corresponding to the filename in the first message with a relationship in the source file server do not match.

[0226] In one possible implementation of this application embodiment, when there is a first message in the second sliding window indicating that no association has been established, it indicates that the file corresponding to the filename in the first message has been missed in synchronizing to the destination file server. At this time, a fifth reconciliation result can be generated based on the first message indicating that no association has been established. The fifth reconciliation result is used to indicate that the destination file server is missing a file corresponding to the filename in the first message indicating that no association has been established.

[0227] In one possible implementation of this application embodiment, when there is a second message in the second sliding window that has not established an association, it indicates that the file corresponding to the filename in the second message is a redundant file in the destination file server, that is, the source file server does not have a file corresponding to the filename in the second message. At this time, a sixth reconciliation result can be generated based on the second message that has not established an association; wherein, the sixth reconciliation result is used to indicate that there is an extra file in the destination file server that corresponds to the filename in the second message that has not established an association.

[0228] Therefore, it can not only identify files that have synchronization errors on the destination file server, but also identify missing and extra files on the destination file server, thus improving the comprehensiveness and completeness of file reconciliation.

[0229] In one possible implementation of this application embodiment, at least one of the following can also be performed based on the reconciliation results:

[0230] The first step is to update the file on the destination file server corresponding to the file in the second message that is related to the above-mentioned file, based on the fourth reconciliation result and the file corresponding to the file in the first message that is related to the above-mentioned file.

[0231] The second step, based on the fifth reconciliation result, is to write the file corresponding to the filename in the first message that has not been associated with the file to the destination file server.

[0232] The third step, based on the sixth reconciliation result, is to delete the file in the destination file server that corresponds to the filename in the second message for which no association has been established.

[0233] This can improve the consistency and integrity of files synchronized or exchanged between the source and target file servers.

[0234] In any embodiment of this application, for a file exchange scenario, the source end can be a source file server, the destination end can be a destination file server, and the structure of the data reconciliation system can be as follows: Figure 7 As shown, Figure 7 The example uses file server A as the source file server, exchange node A as the first exchange node, file server B as the destination file server, and exchange node B as the second exchange node.

[0235] 1. Central control node (i.e. Figure 7 The management node in the middle sends a message from file server A to the first message queue (i.e., the management node in the middle) to exchange node A. Figure 7 The task involves sending a Kafka exchange task from exchange node B to file server B.

[0236] 2. Before starting the exchange task, the central control node can read the current offset1 (index value) of the corresponding topic from Kafka.

[0237] 3. Task A reads files from file server A and uses them in a different format. Figure 9 The format shown is used to write messages to Kafka, with each cell representing one message. A single file can contain multiple messages.

[0238] For example, the file is Figure 9 When stored in Kafka in the format shown, the following information can be included:

[0239] File start flag (startflag): Includes start flag, file relative path, and file update time;

[0240] File binary content: The file is split into multiple binary arrays, each array representing one message;

[0241] End-of-file flag (endflag): Indicates the end of the file stream.

[0242] In this application, messages belonging to the same file can be located based on the file start marker and file end marker.

[0243] 4. Task B parses messages in Kafka and writes the data to the destination file server B.

[0244] 5. After the exchange task between gateway A and gateway B is completed, the central control node reads offset2 (index value) from Kafka.

[0245] 6. Once the central control node detects the completion of the exchange task, it sends a reconciliation task to the reconciliation service. File reconciliation only supports full-data scenarios; in this case, the data between offset2 and offset1 in Kafka represents all the data exchanged.

[0246] 7. Treat the messages between offset2 and offset1 in Kafka as message stream 1.

[0247] 8. Extract files from file server B and... Figure 9 The format shown is written to message queue 3, and message stream 2 is extracted from message queue 3.

[0248] 9. Before formal reconciliation, message stream 1 and message stream 2 are preprocessed to obtain message stream 3 and message stream 4. The preprocessing process includes:

[0249] 1) Merge multiple messages belonging to the same file from message stream 1 or message stream 2, and use a hash algorithm (such as MD5) to hash the binary content of the file, ultimately forming a message in the following format:

[0250]

[0251] 2) Sort the messages formatted in step 1) according to the business timestamp to obtain message stream 3 or message stream 4.

[0252] 10. By comparing message stream 3 and message stream 4, and using time windows and state programming, filenames with inconsistent hash values ​​(such as MD5 values) can be considered to have synchronization errors. Ultimately, the difference files between the source and destination file servers can be obtained, generating reconciliation results.

[0253] Corresponding to the data reconciliation methods provided in the above embodiments, one embodiment of this application also provides a data reconciliation device. Since the data reconciliation device provided in this embodiment corresponds to the data reconciliation methods provided in the above embodiments, the implementation methods of the data reconciliation methods are also applicable to the data reconciliation device provided in this embodiment, and will not be described in detail here.

[0254] Figure 12 This is a schematic diagram of the structure of a data reconciliation device according to an embodiment of this application.

[0255] like Figure 12As shown, the data reconciliation device 1200 may include: a first reading module 1201, a second reading module 1202, and a reconciliation module 1203.

[0256] The first reading module 1201 is used to read the first offset value of the first message queue in response to the detection of the target exchange task; wherein the first offset value is used to indicate the storage location of the last written object in the first message queue; the target exchange task is used to exchange objects between the source and the destination through the first message queue.

[0257] The second reading module 1202 is used to read a second offset value from the first message queue in response to detecting the end of the target exchange task; wherein the second offset value is used to indicate the storage location of the last object in the target exchange task in the first message queue.

[0258] The reconciliation module 1203 is used to reconcile the objects in the source and destination ends that match the target exchange task based on the first offset value and the second offset value, so as to obtain the reconciliation result.

[0259] As one possible implementation of this application embodiment, the object is business data, the source is a source database, and the destination is a destination database; the reconciliation module 1203 is specifically used for: responding to the target exchange task being a full exchange, extracting a first business data stream from a first message queue based on a first offset value and a second offset value; wherein, the offset value corresponding to each first business data in the first business data stream is between the first offset value and the second offset value; extracting a second business data stream from the destination database; wherein, the second business data stream includes all second business data in the destination database; and performing data reconciliation on the first business data stream and the second business data stream to obtain a reconciliation result.

[0260] As one possible implementation of this application, the reconciliation module 1203 is further configured to: extract a third business data stream from the source database; wherein the third business data stream includes all third business data in the source database; and perform data reconciliation between the first business data stream and the third business data stream to obtain a reconciliation result.

[0261] As one possible implementation of this application embodiment, the object is business data, the source is a source database, and the destination is a destination database; the reconciliation module 1203 is specifically used for: responding to the target exchange task being incremental exchange, querying the first business timestamp of the business data matching the first offset value from the first message queue according to the first offset value; querying the second business timestamp of the business data matching the second offset value from the first message queue according to the second offset value; extracting a fourth business data stream from the source database and a fifth business data stream from the destination database according to the first and second business timestamps; performing data reconciliation on the fourth and fifth business data streams to obtain a reconciliation result; wherein, the business timestamps of each fourth business data in the fourth business data stream are between the first and second business timestamps, and the business timestamps of each fifth business data in the fifth business data stream are between the first and second business timestamps.

[0262] As one possible implementation of this application embodiment, the object is business data, the source is a source database, and the destination is a destination database; the data reconciliation device 1200 may further include:

[0263] The first extraction module is used to extract the sixth service data stream from the first message queue in response to the fact that the target exchange task has not ended, based on the first offset value; wherein the offset value corresponding to each sixth service data in the sixth service data stream is greater than the first offset value.

[0264] The second extraction module is used to extract the seventh service data stream from the second message queue according to the third offset value; wherein, the third offset value is used to indicate the storage location of the last written service data in the second message queue before the target exchange task is started, and each seventh service data in the seventh service data stream is the updated service data obtained from the target database in response to the detection of the target exchange task.

[0265] The reconciliation module 1203 is also used to reconcile the data of the sixth business data stream and the seventh business data stream to obtain the reconciliation results.

[0266] As one possible implementation of this application, the reconciliation module 1203 is specifically used to: whenever a set period is reached, use a first sliding window of a first set time length to intercept the first business data stream and the second business data stream; wherein, the first sliding window slides for a second set time length; and reconcile the intercepted first business data and second business data within the first sliding window to obtain a reconciliation result.

[0267] As one possible implementation of this application embodiment, the reconciliation module 1203 is specifically used for: establishing an association between first business data and second business data with matching data identifiers within a first sliding window; comparing the business fields of the associated first business data and second business data, and comparing the business timestamps of the associated first business data and second business data; generating a first reconciliation result based on the associated first business data and second business data when the business fields of the associated first business data and second business data do not match, and / or when the business timestamps of the associated first business data and second business data do not match; wherein, the first reconciliation result is used to indicate that the associated second business data in the destination database does not match the associated first business data in the source database.

[0268] As a possible implementation of this application embodiment, the reconciliation module 1203 is further configured to: generate a second reconciliation result based on the first business data without established association in the first sliding window; and generate a third reconciliation result based on the second business data without established association in the first sliding window; wherein the second reconciliation result is used to indicate that the destination database lacks the first business data without established association; and the third reconciliation result is used to indicate that the destination database has an additional second business data without established association.

[0269] As one possible implementation of this application embodiment, the data reconciliation device 1200 may further include:

[0270] The update module is used to perform at least one of the following:

[0271] Based on the first reconciliation result, the related second business data in the target database is updated according to the related first business data;

[0272] Based on the second reconciliation result, the first business data that has not been associated with the database is written into the target database;

[0273] Based on the third reconciliation result, delete the second business data in the target database that has no established relationship.

[0274] As one possible implementation of this application embodiment, the target exchange task includes a first exchange task and a second exchange task, and the data reconciliation device 1200 may further include:

[0275] The first sending module is used to send a first exchange task to the first exchange node corresponding to the source end in response to the detection of the target exchange task. The first exchange task is used by the first exchange node to obtain the object matching the first exchange task from the source end, sort the objects matching the first exchange task according to the timestamp of the object matching the first exchange task, and write the sorted objects sequentially into the first message queue.

[0276] The second sending module is used to send a second exchange task to the second exchange node corresponding to the destination. The second exchange task is used by the second exchange node to read an object that matches the first exchange task from the first message queue and write it to the destination.

[0277] The second reading module 1202 is specifically used to: read a second offset value from the first message queue in response to detecting the end of the first exchange task and / or the second exchange task.

[0278] In one possible implementation of this application, the object is a file, which is written to a first message queue via multiple messages. The source is a source file server, and the destination is a destination file server. The reconciliation module 1203 is specifically used for: extracting a first message stream from the first message queue based on a first offset value and a second offset value; wherein the offset value corresponding to each message in the first message stream is between the first offset value and the second offset value; generating a second message stream based on the file written to the destination file server; and reconciling the first message stream and the second message stream to obtain a reconciliation result.

[0279] As one possible implementation of this application, the reconciliation module 1203 is specifically used for: merging multiple messages belonging to the same file in the first message stream, performing a hash operation to obtain a first hash value for the same file; generating a first message based on the filename, the first hash value, and the file timestamp corresponding to the same file; sorting each first message according to the file timestamp corresponding to each first message to obtain an updated first message stream; and reconciling the updated first message stream and the second message stream to obtain a reconciliation result.

[0280] As one possible implementation of this application, the reconciliation module 1203 is specifically used for: extracting all target files from the destination file server and performing a hash operation on each target file to obtain a second hash value for each target file; generating a second message for any target file based on the file name, second hash value, and file timestamp corresponding to the target file; and sorting each second message according to the file timestamp corresponding to each second message to obtain a second message stream.

[0281] As one possible implementation of this application embodiment, the reconciliation module 1203 is specifically used for: obtaining a fourth offset value and a fifth offset value, wherein the fourth offset value is used to indicate the storage location of the last message written in the third message queue before the target exchange task starts, and the fifth offset value is used to indicate the storage location of the last message written in the third message queue when the target exchange task ends; extracting an initial message stream from the third message queue according to the fourth offset value and the fifth offset value, wherein each message in the initial message stream is generated based on the file written in the destination file server; merging multiple messages belonging to the same file in the initial message stream and performing a hash operation to obtain a second hash value for the same file; generating a second message according to the file name, the second hash value, and the file timestamp corresponding to the same file; and sorting each second message according to the file timestamp corresponding to each second message to obtain a second message stream.

[0282] As a possible implementation of this application embodiment, the reconciliation module 1203 is specifically used for: whenever a set period is reached, using a second sliding window of a third set time length to intercept the updated first message stream and second message stream; wherein, the second sliding window slides for a fourth set time length; establishing an association between the first message and the second message with matching filenames within the second sliding window; comparing the first hash value of the associated first message and the second hash value of the associated second message, and comparing the file timestamp of the associated first message and the file timestamp of the associated second message; if the first hash value of the associated first message and the second hash value of the associated second message do not match, and / or, the file timestamp of the associated first message and the file timestamp of the associated second message do not match, generating a fourth reconciliation result based on the associated first message and the associated second message; wherein, the fourth reconciliation result is used to indicate that the file in the destination file server corresponding to the filename in the associated second message and the file in the source file server corresponding to the filename in the associated first message do not match.

[0283] As a possible implementation of this application embodiment, the reconciliation module 1203 is further configured to: in response to a first message in the second sliding window indicating that no association has been established, generate a fifth reconciliation result based on the first message indicating that no association has been established; in response to a second message in the second sliding window indicating that no association has been established, generate a sixth reconciliation result based on the second message indicating that no association has been established; wherein, the fifth reconciliation result is used to indicate that the destination file server lacks a file corresponding to the filename in the first message indicating that no association has been established; and the sixth reconciliation result is used to indicate that the destination file server has an extra file corresponding to the filename in the second message indicating that no association has been established.

[0284] The data reconciliation device in this embodiment reads a first offset value from a first message queue in response to the detection of a target exchange task, and reads a second offset value from the first message queue when the target exchange task is detected to be finished. Based on the first and second offset values, it reconciles the objects matching the target exchange task in the source and destination ends to obtain a reconciliation result. Therefore, it is possible to have the central control node store the offset values ​​of the first message queue before and after object exchange (such as data exchange or file exchange), and perform data reconciliation based on these offset values, without needing to store additional batch fields in the destination end, thus reducing intrusion on the destination end.

[0285] To achieve the above embodiments, this application also proposes an electronic device. Figure 13 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes:

[0286] The memory 1301, the processor 1302, and the computer program stored on the memory 1301 and executable on the processor 1302.

[0287] When processor 1302 executes the program, it implements the data reconciliation method provided in any of the above embodiments.

[0288] Furthermore, electronic devices also include:

[0289] Communication interface 1303 is used for communication between memory 1301 and processor 1302.

[0290] The memory 1301 is used to store computer programs that can run on the processor 1302.

[0291] The memory 1301 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0292] The processor 1302 is used to implement the data reconciliation method described in any of the above embodiments when executing the program.

[0293] If the memory 1301, processor 1302, and communication interface 1303 are implemented independently, then the communication interface 1303, memory 1301, and processor 1302 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 13 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0294] Optionally, in a specific implementation, if the memory 1301, processor 1302, and communication interface 1303 are integrated on a single chip, then the memory 1301, processor 1302, and communication interface 1303 can communicate with each other through an internal interface.

[0295] The processor 1302 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0296] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program that, when executed by a processor, implements the data reconciliation method provided in any of the above embodiments.

[0297] To implement the above embodiments, this application also proposes a computer program product that, when executed by an instruction processor, implements the data reconciliation method provided in any of the above embodiments.

[0298] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0299] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0300] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0301] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0302] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0303] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0304] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0305] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A data reconciliation method, characterized by, The method includes: In response to the detection of a target exchange task, a first offset value of the first message queue is read; wherein, the first offset value is used to indicate the storage location of the last written object in the first message queue, the object being business data; the target exchange task is used to exchange objects between the source and the destination through the first message queue, the destination being the target database; In response to the detection that the target exchange task has ended, a second offset value is read from the first message queue; wherein the second offset value is used to indicate the storage location of the last object in the target exchange task in the first message queue; Based on the first offset value and the second offset value, the objects in the source end and the destination end that match the target exchange task are reconciled to obtain the reconciliation result; Based on the first offset value and the second offset value, reconcile the objects in the source and destination ends that match the target exchange task, including: In response to the target exchange task being a full exchange, a first service data stream is extracted from the first message queue based on the first offset value and the second offset value; wherein, the offset value corresponding to each first service data in the first service data stream is between the first offset value and the second offset value; Extract a second business data stream from the target database; wherein the second business data stream includes all the second business data in the target database; Data reconciliation is performed on the first business data stream and the second business data stream to obtain the reconciliation results.

2. The method of claim 1, wherein, The source end is a source database. The reconciliation of objects matching the target exchange task in the source end and the destination end based on the first offset value and the second offset value further includes: Extract a third business data stream from the source database; wherein the third business data stream includes all the third business data in the source database; Data reconciliation is performed on the first business data stream and the third business data stream to obtain the reconciliation results.

3. The method of claim 1, wherein, The source end is the source database; The step of reconciling the objects matching the target exchange task in the source and destination ends based on the first offset value and the second offset value includes: In response to the target exchange task being incremental exchange, the first business timestamp of the business data matching the first offset value is retrieved from the first message queue based on the first offset value. Based on the second offset value, query the second business timestamp of the business data that matches the second offset value from the first message queue; Based on the first business timestamp and the second business timestamp, a fourth business data stream is extracted from the source database, and a fifth business data stream is extracted from the destination database; Data reconciliation is performed on the fourth business data stream and the fifth business data stream to obtain the reconciliation results; The business timestamps of each fourth business data in the fourth business data stream are located between the first business timestamp and the second business timestamp, and the business timestamps of each fifth business data in the fifth business data stream are located between the first business timestamp and the second business timestamp.

4. The method of claim 1, wherein, The source end is the source database; The method further includes: In response to the fact that the target exchange task has not ended, a sixth service data stream is extracted from the first message queue according to the first offset value; wherein, the offset value corresponding to each sixth service data in the sixth service data stream is greater than the first offset value; According to the third offset value, the seventh service data stream is extracted from the second message queue; wherein, the third offset value is used to indicate the storage location of the last service data written in the second message queue before the target exchange task is started, and each seventh service data in the seventh service data stream is updated service data obtained from the target database in response to the detection of the target exchange task; Data reconciliation is performed on the sixth business data stream and the seventh business data stream to obtain the reconciliation results.

5. The method according to any one of claims 2-4, characterized in that, The step of reconciling the first business data stream and the second business data stream to obtain the reconciliation result includes: Whenever a set period is reached, a first sliding window of a first set time length is used to capture the first service data stream and the second service data stream; wherein, the first sliding window slides for a second set time length; The first and second business data captured within the first sliding window are reconciled to obtain the reconciliation results.

6. The method according to claim 5, characterized in that, The step of reconciling the first and second business data captured within the first sliding window to obtain a reconciliation result includes: Establish an association relationship between the first business data and the second business data that match the data identifiers within the first sliding window; The business fields of the first and second business data that are related are compared, and the business timestamps of the first and second business data that are related are also compared. If the business fields of the first and second business data that are related do not match, and / or the business timestamps of the first and second business data that are related do not match, a first reconciliation result is generated based on the first and second business data that are related. The first reconciliation result is used to indicate that the second business data with a relationship in the destination database does not match the first business data with a relationship in the source database.

7. The method according to claim 6, characterized in that, The step of reconciling the first and second business data captured within the first sliding window to obtain a reconciliation result also includes: In response to the existence of first business data without established association within the first sliding window, a second reconciliation result is generated based on the first business data without established association. In response to the existence of second business data without established association within the first sliding window, a third reconciliation result is generated based on the second business data without established association. The second reconciliation result is used to indicate that the first business data for which no association has been established is missing in the target database; The third reconciliation result is used to indicate that the target database contains additional second business data for which no association has been established.

8. The method according to claim 7, characterized in that, The method further includes: Based on the first reconciliation result, the related second business data in the target database is updated according to the related first business data; And / or, Based on the second reconciliation result, the first business data for which no association has been established is written into the destination database; And / or, Based on the third reconciliation result, the second business data in the target database that has not been associated with the target data is deleted.

9. The method according to any one of claims 1-4, characterized in that, The target exchange task includes a first exchange task and a second exchange task; In response to the detection of a target exchange task, the method further includes: Send the first exchange task to the first exchange node corresponding to the source end, wherein the first exchange task is used for the first exchange node to obtain objects matching the first exchange task from the source end, sort the objects matching the first exchange task according to the timestamp of the objects matching the first exchange task, and write the sorted objects sequentially into the first message queue. Send the second exchange task to the second exchange node corresponding to the destination, wherein the second exchange task is used by the second exchange node to read the object matching the first exchange task from the first message queue and write it to the destination; The step of reading a second offset value from the first message queue in response to the detection that the target exchange task has ended includes: In response to the detection that the first exchange task and / or the second exchange task has ended, a second offset value is read from the first message queue.

10. A data reconciliation method, characterized in that, The method includes: In response to the detection of a target exchange task, a first offset value of the first message queue is read; wherein, the first offset value is used to indicate the storage location of the last written object in the first message queue, the object being a file, and the file being written to the first message queue through multiple messages; the target exchange task is used to exchange objects between a source end and a destination end through the first message queue, the source end being a source file server, and the destination end being a destination file server; In response to the detection that the target exchange task has ended, a second offset value is read from the first message queue; wherein the second offset value is used to indicate the storage location of the last object in the target exchange task in the first message queue; Based on the first offset value and the second offset value, the objects in the source end and the destination end that match the target exchange task are reconciled to obtain the reconciliation result; The step of reconciling the objects matching the target exchange task in the source and destination ends based on the first offset value and the second offset value includes: A first message stream is extracted from the first message queue based on the first offset value and the second offset value; wherein the offset value corresponding to each message in the first message stream is between the first offset value and the second offset value; A second message stream is generated based on the file written to the destination file server; The first message stream and the second message stream are reconciled to obtain the reconciliation results.

11. The method according to claim 10, characterized in that, The reconciliation of the first message stream and the second message stream to obtain the reconciliation result includes: After merging multiple messages belonging to the same file in the first message stream, a hash operation is performed to obtain the first hash value of the same file; A first message is generated based on the filename, first hash value, and file timestamp corresponding to the same file; Based on the file timestamps corresponding to each first message, sort each first message to obtain an updated first message stream; The updated first message stream and the second message stream are reconciled to obtain the reconciliation results.

12. The method according to claim 10, characterized in that, The step of generating a second message stream based on the file written to the destination file server includes: Extract all target files from the destination file server and perform a hash operation on each target file to obtain a second hash value for each target file; For any target file, a second message is generated based on the filename, second hash value, and file timestamp corresponding to the target file. Based on the file timestamps corresponding to each second message, the second messages are sorted to obtain the second message stream.

13. The method according to claim 10, characterized in that, The step of generating a second message stream based on the file written to the destination file server includes: Obtain the fourth offset value and the fifth offset value, wherein the fourth offset value is used to indicate the storage location of the last message written in the third message queue before the target exchange task starts, and the fifth offset value is used to indicate the storage location of the last message written in the third message queue when the target exchange task ends. Based on the fourth offset value and the fifth offset value, an initial message stream is extracted from the third message queue, wherein each message in the initial message stream is generated based on the file written in the destination file server; After merging multiple messages belonging to the same file in the initial message stream, a hash operation is performed to obtain the second hash value of the same file; A second message is generated based on the filename, second hash value, and file timestamp of the same file. Based on the file timestamps corresponding to each second message, the second messages are sorted to obtain the second message stream.

14. The method according to claim 11, characterized in that, The reconciliation of the updated first message stream and the second message stream to obtain the reconciliation result includes: Whenever a set period is reached, a second sliding window of a third set time length is used to capture the updated first message stream and the second message stream; wherein, the second sliding window slides for a fourth set time length; Establish an association between the first and second messages whose filenames match within the second sliding window; The first hash value of the first message and the second hash value of the second message that are related are compared, and the file timestamps of the first message and the second message that are related are also compared. If the first hash value of the first message and the second hash value of the second message that are related do not match, and / or if the file timestamp of the first message and the second message that are related do not match, a fourth reconciliation result is generated based on the first message and the second message that are related. The fourth reconciliation result is used to indicate that the file in the destination file server corresponding to the filename in the second message with the association relationship does not match the file in the source file server corresponding to the filename in the first message with the association relationship.

15. The method according to claim 14, characterized in that, The reconciliation of the updated first message stream and the second message stream to obtain the reconciliation result further includes: In response to a first message indicating that no association has been established within the second sliding window, a fifth reconciliation result is generated based on the first message indicating that no association has been established. In response to a second message indicating that no association has been established within the second sliding window, a sixth reconciliation result is generated based on the second message indicating that no association has been established. The fifth reconciliation result is used to indicate that the destination file server is missing a file corresponding to the filename in the first message where no association was established; The sixth reconciliation result is used to indicate that there are additional files in the destination file server that correspond to the filenames in the second message where no association has been established.

16. A data reconciliation device, characterized in that, The device includes: The first reading module is used to read the first offset value of the first message queue in response to the detection of a target exchange task; wherein the first offset value is used to indicate the storage location of the last written object in the first message queue, and the object is business data; the target exchange task is used to exchange objects between the source end and the destination end through the first message queue, and the destination end is the target database; The second reading module is used to read a second offset value from the first message queue in response to detecting the end of the target exchange task; wherein the second offset value is used to indicate the storage location of the last object in the target exchange task in the first message queue; The reconciliation module is used to reconcile the objects matching the target exchange task in the source end and the destination end according to the first offset value and the second offset value, so as to obtain the reconciliation result; Based on the first offset value and the second offset value, reconcile the objects in the source and destination ends that match the target exchange task, including: In response to the target exchange task being a full exchange, a first service data stream is extracted from the first message queue based on the first offset value and the second offset value; wherein, the offset value corresponding to each first service data in the first service data stream is between the first offset value and the second offset value; Extract a second business data stream from the target database; wherein the second business data stream includes all the second business data in the target database; Data reconciliation is performed on the first business data stream and the second business data stream to obtain the reconciliation results.

17. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the data reconciliation method as described in any one of claims 1-15.

18. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data reconciliation method as described in any one of claims 1-17.

Images