Cross-partition reconciliation method and device, storage medium and computer device

Abstract

The cross-partition reconciliation method, apparatus, storage medium, and computer equipment provided in this application effectively solve the "false anomaly" problem caused by data warehousing delays in single-partition reconciliation by introducing the reconciliation results of historical data partitions for cross-partition comparison. Compared with the traditional single-partition reconciliation mode, the cross-partition comparison mechanism of this application, through the core logic of cross-validation of multi-partition data, can effectively filter out false anomalies caused by non-business factors such as data delays and partition synchronization time differences, significantly reducing the false alarm rate of reconciliation results, achieving a "noise reduction" effect in reconciliation, making inconsistencies in real business data more accurately highlighted, improving the credibility of reconciliation results and their reference value for business decisions, reducing the workload of manual review, and reducing the risk of missing or misjudging real abnormal transactions, thereby meeting the large-scale and time-sensitive reconciliation needs in scenarios such as finance and e-commerce.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a cross-regional reconciliation method, apparatus, storage medium and computer equipment. Background Technology

[0002] In financial and e-commerce scenarios, transaction data is typically stored in partitions based on time (e.g., by day, by hour) to improve data query and processing efficiency. However, traditional reconciliation methods are often limited to comparisons within a single data partition, meaning they only verify data between the first and second data sources within the current partition. (Illustratively, for example...) Figure 1 As shown, Figure 1 This is a flowchart illustrating single-partition alignment in existing technologies. Figure 1 In this context, hm=1500 indicates that the data partitioning time is 3 PM, and the two different business systems will start the data inbound task after 3 PM. Due to the latency differences in data inbound between different business systems, reconciliation of a single partition is prone to "false anomalies": for example... Figure 1 In business database 1, the inbound execution time is 15:10, and the corresponding partition's data consists of data from 14:00 to 15:10 plus data from the previous partition. Business database 2's inbound execution time is 15:30, and the corresponding partition's data also consists of data from 14:00 to 15:30 plus data from the previous partition. When comparing the data for the 15:00 partition, because business database 2 has 20 minutes more data than business database 1, reconciliation based on the current partition might misclassify the extra data as abnormal. Furthermore, some data noise (such as temporary test data or duplicated invalid data) might also be misidentified as abnormal in single-partition reconciliation, affecting the accuracy of the reconciliation results.

[0003] Therefore, this reconciliation method, which is limited to a single partition, not only increases the workload of manual review, but may also lead to the omission or misjudgment of real abnormal transactions, making it difficult to meet the needs of large-scale and time-sensitive business reconciliation. Summary of the Invention

[0004] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the technical deficiency of the existing single-partition reconciliation method, which not only increases the workload of manual review but may also lead to the omission or misjudgment of genuine abnormal transactions, making it difficult to meet the technical deficiencies of large-scale, high-timeliness business reconciliation needs.

[0005] This application provides a cross-regional reconciliation method, the method comprising:

[0006] Determine the current data partition, where the current data partition is a time partition;

[0007] Obtain the current reconciliation result, which is obtained by comparing the data in the current data partition with the data in the first data source and the second data source. The data in the first data source and the second data source come from different business systems and use the same partition span.

[0008] Obtain at least one historical reconciliation result, wherein each historical reconciliation result is obtained by comparing the data of the first data source with the data of the second data source in at least one historical data partition prior to the current data partition;

[0009] The current reconciliation result is compared with at least one of the historical reconciliation results, and the final reconciliation result of the current data partition is determined based on the first comparison result.

[0010] Optionally, obtaining at least one historical reconciliation result includes:

[0011] Determine the delay in the inbound data of the first data source and / or the second data source;

[0012] Based on the source data inbound delay, a selection strategy for historical data partitions used to obtain historical reconciliation results is determined. The selection strategy includes the number of historical reconciliation results to be obtained and the span of the historical data partition relative to the current data partition.

[0013] Based on the selection strategy, obtain the historical reconciliation results of at least one historical data partition prior to the current data partition.

[0014] Optionally, determining the source data inbound delay of the first data source and / or the second data source includes:

[0015] Get the first completion time of the inbound task of the first data entering the first data source from the corresponding business system, calculate the first deviation between the first completion time and the theoretical partition time window of the first data, and determine the source data inbound delay of the first data source based on the first deviation.

[0016] And / or,

[0017] The second completion time of the inbound task of the second data entering the second data source from the corresponding business system is obtained. The second deviation between the second completion time and the theoretical partition time window of the second data is calculated. The source data inbound delay of the second data source is determined based on the second deviation.

[0018] Optionally, the step of determining the selection strategy for the historical data partition used to obtain historical reconciliation results based on the source data inbound delay includes:

[0019] The source data inbound delay is compared with the time length of a standard partition to obtain a second comparison result;

[0020] Based on the second comparison result, the quantity and / or the span are dynamically adjusted, and a selection strategy for historical data partitions used to obtain historical reconciliation results is determined according to the adjusted quantity and / or span.

[0021] Optionally, the current reconciliation result includes current abnormal data and current noisy data, and each of the historical reconciliation results includes historical abnormal data and historical noisy data;

[0022] The step of comparing the current reconciliation result with at least one of the historical reconciliation results, and determining the final reconciliation result of the current data partition based on the comparison result, includes:

[0023] The current abnormal data and the current noise data are combined with at least one of the historical abnormal data and the historical noise data to perform a set operation to obtain the final reconciliation result.

[0024] Optionally, the set operation is an intersection operation.

[0025] Optionally, the method further includes:

[0026] Count the number of abnormal data in the final reconciliation results;

[0027] If the number of abnormal data exceeds a preset threshold, an abnormal alarm will be issued.

[0028] This application also provides a cross-regional reconciliation device, including:

[0029] The time partition determination module is used to determine the current data partition, wherein the current data partition is a time partition;

[0030] The first result acquisition module is used to acquire the current reconciliation result. The current reconciliation result is obtained by comparing the data in the current data partition with the data in the first data source and the second data source. The data in the first data source and the second data source come from different business systems and use the same partition span.

[0031] The second result acquisition module is used to acquire at least one historical reconciliation result, wherein each historical reconciliation result is obtained by comparing the data of the first data source with the data of the second data source in at least one historical data partition before the current data partition;

[0032] The cross-partition comparison module is used to compare the current reconciliation result with at least one of the historical reconciliation results, and determine the final reconciliation result of the current data partition based on the first comparison result.

[0033] This application also provides a computer-readable storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the cross-partition reconciliation method as described in any of the above embodiments.

[0034] This application also provides a computer device, including: one or more processors, and memory;

[0035] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the cross-partition reconciliation method as described in any of the above embodiments.

[0036] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0037] The cross-partition reconciliation method, apparatus, storage medium, and computer equipment provided in this application effectively solve the "false anomaly" problem caused by data entry delays in single-partition reconciliation by introducing the reconciliation results of historical data partitions for cross-partition comparison. For example, when the entry delay of the first data source is 20 minutes and the entry delay of the second data source is 30 minutes, the reconciliation result of the current data partition may contain data that has not been synchronized due to the delay. However, by retrieving the results of the previous one or two historical data partitions and performing relevant calculations, noisy data that only appears temporarily in the current partition can be filtered out. Compared to the traditional single-partition reconciliation model, the cross-partition comparison mechanism of this application, through the core logic of cross-validation of multi-partition data, can effectively filter out false anomalies caused by non-business factors such as data delays and partition synchronization time differences, significantly reduce the false alarm rate of reconciliation results, achieve the effect of "noise reduction" in reconciliation, make the inconsistency of real business data more accurately stand out, improve the credibility of reconciliation results and the reference value for business decisions, reduce the workload of manual review, reduce the risk of real abnormal transactions being missed or misjudged, and thus meet the large-scale and time-sensitive reconciliation needs in scenarios such as finance and e-commerce. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A flowchart illustrating single-partition alignment in existing technologies;

[0040] Figure 2A flowchart illustrating a cross-regional reconciliation method provided in this application embodiment;

[0041] Figure 3 A schematic diagram illustrating the process of comparing the data of the two partitions provided in this embodiment of the application;

[0042] Figure 4 A schematic diagram of a cross-regional reconciliation device provided in this application embodiment;

[0043] Figure 5 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0045] In one embodiment, such as Figure 2 As shown, Figure 2 This application provides a flowchart illustrating a cross-regional reconciliation method according to an embodiment of the present application; the present application provides a cross-regional reconciliation method, which may include:

[0046] S110: Determine the current data partition.

[0047] In this step, the current data partition is a time partition, which can be specifically divided according to the time granularity preset by the business system, such as by hour, by day, or by week. Taking the transaction data reconciliation of an e-commerce platform as an example, if the business system uses an hourly time partition span, then the current data partition can be determined as the time interval from 14:00 to 15:00 of the current day; if it uses a daily span, then the current data partition is the time interval from 00:00 to 24:00 of the current day.

[0048] Furthermore, when determining the current data partition, the corresponding time window parameters, including the start timestamp, end timestamp, and partition identifier code, can be obtained simultaneously to accurately locate the data to be compared between the first data source and the second data source within that partition.

[0049] S120: Get the current reconciliation results.

[0050] In this step, after determining the current data partition through S110, this application can also obtain the current reconciliation result, which is obtained by comparing the data of the first data source and the second data source within the current data partition.

[0051] The first and second data sources come from different business systems. For example, the first data source is the transaction order system database of the e-commerce platform, and the second data source is the transaction flow database of the third-party payment system. Both use the same time partition span to ensure the consistency of the time range of the data to be compared.

[0052] The comparison process between the first and second data sources can specifically include: First, extracting all transaction records within the current data partition from the first data source to generate a first data set, where each record contains core fields such as order number, transaction amount, transaction time, and payment method; then, extracting all transaction records within the same time partition from the second data source to generate a second data set, where each record contains corresponding fields such as transaction number, transaction amount, transaction time, and associated order number; next, using the order number (or associated order number) as the matching key, performing a full comparison between the two data sets, marking records that fail to match as current abnormal data, and marking test records that appear 3 or more times and have a transaction amount of 0 as current noisy data, and finally integrating the current abnormal data and the current noisy data to form the current reconciliation result.

[0053] S130: Obtain at least one historical reconciliation result.

[0054] In this step, after determining the current data partition through S110, this application can also obtain at least one historical reconciliation result. Each historical reconciliation result is obtained by comparing the data of the first data source and the second data source in at least one historical data partition before the current data partition.

[0055] Understandably, since the latency of data entering the warehouse varies across different business data sources, this application can determine the number of historical reconciliation results to be obtained based on the latency of data from different data sources. For example, for some source data with significant latency, this application can specifically increase the span of the partition to better reduce false alarms caused by data latency, while for source data with less latency, the span can be appropriately reduced to balance reconciliation efficiency and accuracy.

[0056] Specifically, if the inbound delay of the first data source is stable within 30 minutes and the inbound delay of the second data source is stable within 45 minutes, and the time span of the current data partition is 1 hour (i.e. 60 minutes), then the historical data partition selection strategy can be determined to retrieve the reconciliation results of the previous historical partition (i.e., the partition of the previous hour) of the current partition; if the inbound delay of the second data source suddenly increases to 90 minutes, then the selection strategy can be dynamically adjusted to retrieve the reconciliation results of the previous 2 historical partitions to ensure coverage of the data range with delayed arrival.

[0057] It should be noted that when obtaining historical reconciliation results, this application must follow the same generation logic as the current reconciliation results, that is, to perform a full comparison of the data from the first data source and the second data source within the historical data partition, mark historical abnormal data and historical noisy data, and form a structured set of historical reconciliation results.

[0058] S140: Compare the current reconciliation result with at least one historical reconciliation result, and determine the final reconciliation result of the current data partition based on the first comparison result.

[0059] In this step, after obtaining the current reconciliation result through S120 and at least one historical reconciliation result through S130, this application can compare the current reconciliation result with at least one historical reconciliation result to obtain a first comparison result. This application can determine the final reconciliation result of the current data partition based on the first comparison result.

[0060] In this application, when comparing the current reconciliation result with at least one historical reconciliation result, different comparison dimensions can be selected based on the number of historical reconciliation results and the distribution characteristics of abnormal data in each historical reconciliation result. For example, if only one historical reconciliation result is obtained, the current abnormal data is directly matched one-to-one with that historical abnormal data; if two or more historical reconciliation results are obtained, all historical abnormal data are first merged and deduplicated to generate a total set of historical abnormal data, and then compared with the current abnormal data.

[0061] The specific comparison process refers to the noise reduction process, which can be achieved in various ways, such as by building a dynamic noise filtering rule base. The rule base includes preset rules based on business scenarios and self-learning rules based on historical data: preset rules include "records with a transaction amount of 0 and no actual associated orders are judged as test noise" and "records with the same user ID submitted more than 10 times within 5 minutes are judged as duplicate noise"; self-learning rules analyze the historical reconciliation results of the past 30 days, statistically analyze the characteristic distribution of noise data (such as the repetition frequency threshold and the range of amount fluctuation), and automatically update the rule thresholds using a decision tree model. During the comparison, the current abnormal data can first be matched with the rule base to filter out noise such as temporary test data and duplicate invalid data that meet the rules. Then, the remaining abnormal data can be correlated with the abnormal data in the historical reconciliation results. For example, if a transaction record (order number A) in the current abnormal data has been matched in the reconciliation results of the historical partition (i.e., the transaction record of order number A in the historical partition has been received), then the record is determined to be a "false anomaly with delayed arrival" and excluded. Finally, the real abnormal data that has not been filtered is retained to form the final reconciliation result. In addition, this application can also use a text denoising method based on cosine similarity to calculate the similarity between unstructured fields (such as transaction notes) in the current reconciliation result and the corresponding fields in the historical noise data. If the similarity exceeds 90%, it is determined to be the same type of noise and filtered out, further improving the denoising accuracy.

[0062] In the above embodiments, by introducing reconciliation results from historical data partitions for cross-partition comparison, the problem of "false anomalies" caused by data warehousing delays in single-partition reconciliation is effectively solved. For example, when the warehousing delay of the first data source is 20 minutes and that of the second data source is 30 minutes, the reconciliation result of the current data partition may contain data that is not synchronized due to the delay. However, by retrieving the results of the previous one or two historical data partitions and performing relevant calculations, noisy data that only appears temporarily in the current partition can be filtered out. Compared with the traditional single-partition reconciliation mode, the cross-partition comparison mechanism of this application, through the core logic of cross-validation of multi-partition data, can effectively filter out false anomalies caused by non-business factors such as data delays and partition synchronization time differences, significantly reduce the false alarm rate of reconciliation results, achieve a "noise reduction" effect in reconciliation, make the inconsistency of real business data more accurately stand out, improve the credibility of reconciliation results and the reference value for business decisions, reduce the workload of manual review, reduce the risk of real abnormal transactions being missed or misjudged, and thus meet the large-scale and time-sensitive reconciliation needs in scenarios such as finance and e-commerce.

[0063] In one embodiment, obtaining at least one historical reconciliation result in S130 may include:

[0064] S131: Determine the delay in the inbound data of the first data source and / or the second data source.

[0065] S132: Based on the source data inbound delay, determine the selection strategy for historical data partitions used to obtain historical reconciliation results. The selection strategy includes the number of historical reconciliation results to be obtained and the span of the historical data partition relative to the current data partition.

[0066] S132: According to the selection strategy, obtain the historical reconciliation results of at least one historical data partition prior to the current data partition.

[0067] In this embodiment, when obtaining at least one historical reconciliation result, the data entry delay of the first data source and / or the second data source can be determined first. Specifically, this can be achieved by real-time monitoring of the data source's entry logs. For example, recording the time difference between each piece of data being generated by the business system and written to the data warehouse, calculating the average delay, maximum delay, and delay variance over seven consecutive days, and forming a delay characteristic profile. Taking a core transaction system in the financial industry as an example, if the average entry delay of the first data source (bank core system) is 25 minutes, the maximum delay is 40 minutes, and the delay variance is 5; and the average entry delay of the second data source (UnionPay clearing system) is 35 minutes, the maximum delay is 60 minutes, and the delay variance is 8, then the subsequent selection strategy can be determined by comprehensively considering the delay characteristics of both.

[0068] Next, this application can determine the selection strategy for historical data partitions used to obtain historical reconciliation results based on the source data inbound delay. This selection strategy includes the number of historical reconciliation results to be obtained and the span of the historical data partition relative to the current data partition. For example, if the time span of the current data partition is 1 hour, the maximum delay of the first data source is 40 minutes, and the maximum delay of the second data source is 60 minutes, then the selection strategy can be determined as follows: the number of historical reconciliation results to be obtained is 2, and the span of the historical data partition relative to the current data partition is the partition of the previous 2 consecutive hours (i.e., when the current partition is 14:00-15:00, the historical partition is 12:00-13:00 and 13:00-14:00), ensuring coverage of the maximum delay range of 60 minutes of the second data source; if the maximum delay of the second data source drops to 30 minutes, the selection strategy can be dynamically adjusted to: the number of historical reconciliation results to be obtained is 1, and the historical data partition is the partition of the previous 1 hour, in order to reduce the amount of data retrieved and improve reconciliation efficiency.

[0069] Finally, this application can obtain the historical reconciliation results of at least one historical data partition prior to the current data partition based on the selection strategy. Specifically, the acquisition process can quickly locate the target historical partition using the partition index of the distributed data warehouse. For example, it can call the Hive partition query statement "SELECT * FROM reconciliation result table WHERE partition identifier IN('2024052012', '2024052013')" and store the returned historical reconciliation results in JSON format in a memory cache. Each historical reconciliation result must include metadata such as the historical partition identifier, a list of historical abnormal data, a list of historical noisy data, and a generation timestamp, so that it can be quickly compared with the current reconciliation result later.

[0070] In one embodiment, determining the source data inbound delay of the first data source and / or the second data source in S131 may include:

[0071] S1311: Obtain the first completion time of the inbound task of the first data entering the first data source from the corresponding business system, calculate the first deviation between the first completion time and the theoretical partition time window of the first data, and determine the source data inbound delay of the first data source based on the first deviation.

[0072] S1312: and / or, obtain the second completion time of the inbound task of the second data entering the second data source from the corresponding business system, calculate the second deviation between the second completion time and the theoretical partition time window of the second data, and determine the source data inbound delay of the second data source based on the second deviation.

[0073] In this embodiment, when determining the source data warehousing delay, this application takes the transaction order system (first data source) and third-party payment system (second data source) of an e-commerce platform as examples. The first data is the order record in the transaction order system, and its theoretical partition time window is the start timestamp to the end timestamp of the current data partition (e.g., 14:00:00 to 15:00:00 on 2024-05-20). When the actual generation time of a certain order record (order number B) is 14:45:00 on 2024-05-20, it falls within the current theoretical partition time window. However, the completion time of the warehousing task of writing the record from the business system to the first data source is 15:10:00 on 2024-05-20. At this time, the first deviation between the first completion time (15:10:00) and the end time of the theoretical partition time window (15:00:00) is calculated to be 10 minutes, that is, the warehousing delay of the order record is 10 minutes. By performing the above deviation calculation on all order records of the first data source within the current data partition, and statistically analyzing the average delay, maximum delay, and the percentage of records with delays exceeding a preset threshold (such as 20 minutes), the source data inbound delay of the first data source can be fully determined.

[0074] Similarly, the second data source is the transaction record of a third-party payment system. Assuming that the theoretical time window for a certain transaction record (associated with order number B) is also 14:00:00 to 15:00:00, its actual generation time is 2024-05-20 14:50:00, and the warehouse entry task completion time is 2024-05-20 15:25:00, the second deviation between the second completion time and the theoretical end time of the time window is calculated to be 25 minutes. This means the warehouse entry delay for this transaction record is 25 minutes. After statistically analyzing the deviations of all transaction records within the second data source, the warehouse entry delay of the source data from the second data source can be obtained.

[0075] This application, by separately determining the inbound latency of two data sources, enables a more precise strategy for selecting historical data partitions, avoiding reconciliation errors caused by excessive latency of a single data source.

[0076] In one embodiment, determining the selection strategy for the historical data partition used to obtain historical reconciliation results based on the source data inbound delay in step S132 may include:

[0077] S1321: Compare the source data warehouse entry delay with a standard partition time length to obtain a second comparison result.

[0078] S1322: Based on the second comparison result, dynamically adjust the quantity and / or the span, and determine the selection strategy for historical data partitions used to obtain historical reconciliation results according to the adjusted quantity and / or span.

[0079] In this embodiment, the standard partition time length refers to the preset time span of the current data partition, such as 1 hour or 2 hours. By comparing the source data inbound latency (such as maximum latency and average latency) with this standard length, the impact of latency on the coverage of a single partition can be clearly identified. Taking a standard partition time length of 1 hour (60 minutes) as an example, if the maximum inbound latency of the first data source is 40 minutes and the maximum inbound latency of the second data source is 70 minutes, the second comparison result is "the maximum latency of the second data source exceeds the standard partition time length by 10 minutes"; if the maximum latency of both is 30 minutes, the second comparison result is "neither exceeds the standard partition time length".

[0080] Based on the second comparison result mentioned above, this application can dynamically adjust the number of historical reconciliation results obtained and the historical data partition span. For example, when the second comparison result shows that the maximum delay of a certain data source exceeds the standard partition time length, the span of the historical data partition needs to be increased: if the standard partition is 1 hour and the maximum delay of the second data source is 70 minutes (exceeding 10 minutes), then the historical data partition span is adjusted from "the first partition" to "the first 2 partitions" to ensure that the delayed data can be covered by the historical partition; if the second comparison result shows that the delay of all data sources does not exceed the standard length, but the average delay is close to 80% of the standard length (such as a standard of 1 hour with an average delay of 48 minutes), then the number of historical reconciliation results obtained is increased from 1 to 2, and the risk of delay misjudgment is reduced through multi-partition cross-validation.

[0081] Conversely, if subsequent monitoring reveals that the maximum latency of the second data source has decreased to 50 minutes (not exceeding the standard 1 hour), the span can be reverted to "the previous partition" to reduce data retrieval and improve efficiency. Ultimately, based on the adjusted quantity and span, a specific historical data partition selection strategy is determined, such as "retrieve the reconciliation results of the two consecutive historical partitions before the current partition, with each partition spanning 1 hour," thereby achieving dynamic adaptation between the selection strategy and the data source latency.

[0082] In one embodiment, the current reconciliation result includes current abnormal data and current noise data, and each of the historical reconciliation results includes historical abnormal data and historical noise data.

[0083] S140 compares the current reconciliation result with at least one of the historical reconciliation results, and determines the final reconciliation result of the current data partition based on the comparison result, which may include:

[0084] The current abnormal data and the current noise data are combined with at least one of the historical abnormal data and the historical noise data to perform a set operation to obtain the final reconciliation result.

[0085] In this embodiment, when comparing the current reconciliation result with at least one historical reconciliation result, precise filtering can be achieved through set operations. For example, the current abnormal data is denoted as set A, and the current noisy data is denoted as set N; the historical abnormal data in each historical reconciliation result is denoted as sets B1, B2…Bn (n≥1), and the historical noisy data is denoted as sets M1, M2…Mn. First, the union operation is performed on all historical noisy data sets to obtain the total historical noise set M_total = M1∪M2∪…∪Mn; then, the intersection operation is performed on the current noisy data set N and M_total to obtain N∩M_total. If a piece of data exists in both N and M_total, it is determined to be "high-frequency repetitive noise" and directly filtered.

[0086] Next, this application performs a union operation on the historical abnormal data set to obtain the historical abnormal set B_total = B1∪B2∪…∪Bn; it then performs a difference operation on the current abnormal data set A and B_total to obtain A-B_total, which represents new abnormal data in the current abnormal data that does not appear in the historical abnormal set; simultaneously, it calculates the difference between B_total and A, B_total - A, which represents residual abnormal data in the historical abnormal data that has been resolved but not eliminated in the current abnormal data. Finally, it merges A-B_total and B_total - A to obtain the final abnormal data set; and then combines this with the filtered noise data to generate a final reconciliation result that includes categories such as "new abnormal data," "residual abnormal data," and "filtered noise data."

[0087] Taking an e-commerce scenario as an example, the current abnormal data set A contains order C (unmatched transaction history) and order D (inconsistent amount), the historical abnormal data set B1 contains order C (unmatched in the previous partition) and order E (resolved), and the historical noise set M1 contains "test order F". After set operations, A-B_total = {D} (new abnormality), B_total-A = {E} (residual abnormality), and N∩M_total = {F} (high-frequency noise). The final reconciliation result clearly presents the three types of data, making it easier for business personnel to handle them in a targeted manner.

[0088] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the process of comparing the partitioned data of the two entities, as provided in an embodiment of this application; the set operation is the intersection operation.

[0089] In this embodiment, when comparing the current reconciliation result with at least one historical reconciliation result, the intersection operation of the current reconciliation result and at least one historical reconciliation result can be performed. This can remove the noisy data in the two partitions and thus obtain the real abnormal data.

[0090] Figure 3 The result data 1 is Figure 1 In the example solution, result data 1 contains abnormal data and noisy data 1. If the current data partition hm=1600, after completing a single partition reconciliation, result data 2 will be obtained, which contains abnormal data and noisy data 2. Noisy data 2 may contain delayed transaction data caused by temporary network fluctuations. Performing an intersection operation on result data 1 and result data 2 extracts the abnormal data that exists simultaneously in both result data. Temporary interference data in noisy data 1 and noisy data 2 will be filtered out because they do not occur simultaneously. The final intersection data is the truly abnormal data verified across partitions.

[0091] For example, in result data 1, the abnormal data includes order G (unmatched transaction), and in noisy data 1, it includes "temporary test order H"; in result data 2, the abnormal data includes order G, and in noisy data 2, it includes "network delay order I". Taking the intersection, only order G is retained, effectively eliminating noise data like H and I that only appear once, ensuring the accuracy of the final abnormal data. This cross-partition comparison method based on intersection operations can utilize data characteristics from historical reconciliation results to accurately distinguish between persistent anomalies and occasional noise, further reducing the risk of misjudging abnormal transactions and improving the reliability of reconciliation results.

[0092] In one embodiment, the method may further include:

[0093] S150: Count the number of abnormal data in the final reconciliation results.

[0094] S151: If the number of abnormal data exceeds a preset threshold, an abnormal alarm will be issued.

[0095] In this embodiment, after determining the final reconciliation result of the current data distinction, this application can also count the number of abnormal data in the final reconciliation result and compare the number of abnormal data with a preset number threshold to determine whether the abnormal data exceeds the threshold.

[0096] The preset threshold for the number of records in this application can be dynamically configured according to the risk level of different business scenarios. For example, the threshold for daily reconciliation of e-commerce platforms is set at 50 records, which can be adjusted to 200 records during peak sales periods (such as "Double Eleven") due to increased transaction volume. When counting the number of abnormal data in the final reconciliation results, it is also possible to distinguish between the sum of "new abnormal data" and "residual abnormal data" to ensure coverage of all unresolved reconciliation issues. If the final number of abnormal data in the current data partition (14:00-15:00) of an e-commerce platform is 60 records, exceeding the daily threshold of 50 records, an abnormality alarm mechanism will be triggered.

[0097] Furthermore, the alerting methods in this application may include sending real-time SMS / email notifications to the reconciliation system administrator, displaying a red warning indicator on the platform monitoring screen, and attaching a detailed list of abnormal data (such as abnormal order number, associated data source, and abnormality type). After receiving the alert, the administrator can quickly locate the problem: if 45 out of 60 abnormalities are "new abnormal data" and are concentrated in the missing transaction records of the third-party payment system, it can be initially determined that there are fluctuations in the payment system interface, and the payment service provider should be contacted in a timely manner for investigation; if 15 "residual abnormal data" are orders with unresolved amount discrepancies in three consecutive partitions, it is necessary to work with the business department to verify the original vouchers of the orders to avoid long-term outstanding accounts and the resulting financial risks.

[0098] Furthermore, this application supports dynamic classification of alarm levels: when the number of anomalies exceeds a threshold of 100% (e.g., a daily threshold of 50, but an actual threshold of 100), the highest-level emergency alarm is triggered and directly pushed to the department head; if the number exceeds the threshold by less than 50%, a normal alarm is triggered, which is handled routinely by operations and maintenance personnel. Through this alarm mechanism based on quantity thresholds, this application can achieve timely detection and closed-loop processing of reconciliation anomalies, ensuring the accuracy of cross-regional reconciliation and the security of business funds.

[0099] The cross-regional reconciliation device provided in the embodiments of this application is described below. The cross-regional reconciliation device described below can be referred to in correspondence with the cross-regional reconciliation method described above.

[0100] In one embodiment, such as Figure 4 As shown, Figure 4 This application provides a schematic diagram of the structure of a cross-partition reconciliation device according to an embodiment of the present application. The present application also provides a cross-partition reconciliation device, which may include a time partition determination module 210, a first result acquisition module 220, a second result acquisition module 230, and a cross-partition comparison module 240, specifically including the following:

[0101] The time partition determination module 210 is used to determine the current data partition, wherein the current data partition is a time partition.

[0102] The first result acquisition module 220 is used to acquire the current reconciliation result, which is obtained by comparing the data in the current data partition with the data in the first data source and the second data source. The data in the first data source and the second data source come from different business systems and use the same partition span.

[0103] The second result acquisition module 230 is used to acquire at least one historical reconciliation result, wherein each historical reconciliation result is obtained by comparing the data of the first data source with the data of the second data source in at least one historical data partition before the current data partition.

[0104] The cross-partition comparison module 240 is used to compare the current reconciliation result with at least one of the historical reconciliation results, and determine the final reconciliation result of the current data partition based on the first comparison result.

[0105] In the above embodiments, by introducing reconciliation results from historical data partitions for cross-partition comparison, the problem of "false anomalies" caused by data warehousing delays in single-partition reconciliation is effectively solved. For example, when the warehousing delay of the first data source is 20 minutes and that of the second data source is 30 minutes, the reconciliation result of the current data partition may contain data that is not synchronized due to the delay. However, by retrieving the results of the previous one or two historical data partitions and performing relevant calculations, noisy data that only appears temporarily in the current partition can be filtered out. Compared with the traditional single-partition reconciliation mode, the cross-partition comparison mechanism of this application, through the core logic of cross-validation of multi-partition data, can effectively filter out false anomalies caused by non-business factors such as data delays and partition synchronization time differences, significantly reduce the false alarm rate of reconciliation results, achieve a "noise reduction" effect in reconciliation, make the inconsistency of real business data more accurately stand out, improve the credibility of reconciliation results and the reference value for business decisions, reduce the workload of manual review, reduce the risk of real abnormal transactions being missed or misjudged, and thus meet the large-scale and time-sensitive reconciliation needs in scenarios such as finance and e-commerce.

[0106] In one embodiment, this application also provides a computer-readable storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the cross-partition reconciliation method as described in any of the above embodiments.

[0107] In one embodiment, this application also provides a computer device, including: one or more processors, and memory.

[0108] The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the cross-partition reconciliation method as described in any of the above embodiments.

[0109] Indicatively, such as Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device 300 provided in an embodiment of this application. The computer device 300 can be provided as a server. (Refer to...) Figure 5The computer device 300 includes a processing component 302, which further includes one or more processors, and memory resources represented by memory 301 for storing instructions, such as application programs, that can be executed by the processing component 302. The application programs stored in memory 301 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 302 is configured to execute instructions to perform the cross-partition reconciliation method of any of the above embodiments.

[0110] The computer device 300 may also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate on an operating system stored in memory 301, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0111] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0112] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0113] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0114] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cross-regional reconciliation method, characterized in that, The method includes: Determine the current data partition, where the current data partition is a time partition; Obtain the current reconciliation result, which is obtained by comparing the data in the current data partition with the data in the first data source and the second data source. The data in the first data source and the second data source come from different business systems and use the same partition span. Obtain at least one historical reconciliation result, wherein each historical reconciliation result is obtained by comparing the data of the first data source with the data of the second data source in at least one historical data partition prior to the current data partition; The current reconciliation result is compared with at least one of the historical reconciliation results, and the final reconciliation result of the current data partition is determined based on the first comparison result.

2. The cross-regional reconciliation method according to claim 1, characterized in that, Obtaining at least one historical reconciliation result includes: Determine the delay in the inbound data of the first data source and / or the second data source; Based on the source data inbound delay, a selection strategy for historical data partitions used to obtain historical reconciliation results is determined. The selection strategy includes the number of historical reconciliation results to be obtained and the span of the historical data partition relative to the current data partition. Based on the selection strategy, obtain the historical reconciliation results of at least one historical data partition prior to the current data partition.

3. The cross-regional reconciliation method according to claim 2, characterized in that, The determination of the source data inbound delay of the first data source and / or the second data source includes: Get the first completion time of the inbound task of the first data entering the first data source from the corresponding business system, calculate the first deviation between the first completion time and the theoretical partition time window of the first data, and determine the source data inbound delay of the first data source based on the first deviation. And / or, The second completion time of the inbound task of the second data entering the second data source from the corresponding business system is obtained. The second deviation between the second completion time and the theoretical partition time window of the second data is calculated. The source data inbound delay of the second data source is determined based on the second deviation.

4. The cross-regional reconciliation method according to claim 2, characterized in that, The selection strategy for historical data partitions used to obtain historical reconciliation results, based on the source data inbound delay, includes: The source data inbound delay is compared with the time length of a standard partition to obtain a second comparison result; Based on the second comparison result, the quantity and / or the span are dynamically adjusted, and a selection strategy for historical data partitions used to obtain historical reconciliation results is determined according to the adjusted quantity and / or span.

5. The cross-regional reconciliation method according to claim 1, characterized in that, The current reconciliation result includes current abnormal data and current noisy data, and each of the historical reconciliation results includes historical abnormal data and historical noisy data; The step of comparing the current reconciliation result with at least one of the historical reconciliation results, and determining the final reconciliation result of the current data partition based on the comparison result, includes: The current abnormal data and the current noise data are combined with at least one of the historical abnormal data and the historical noise data to perform a set operation to obtain the final reconciliation result.

6. The cross-regional reconciliation method according to claim 5, characterized in that, The set operation is the intersection operation.

7. The cross-regional reconciliation method according to any one of claims 1-6, characterized in that, The method further includes: Count the number of abnormal data in the final reconciliation results; If the number of abnormal data exceeds a preset threshold, an abnormal alarm will be issued.

8. A cross-regional reconciliation device, characterized in that, include: The time partition determination module is used to determine the current data partition, wherein the current data partition is a time partition; The first result acquisition module is used to acquire the current reconciliation result. The current reconciliation result is obtained by comparing the data in the current data partition with the data in the first data source and the second data source. The data in the first data source and the second data source come from different business systems and use the same partition span. The second result acquisition module is used to acquire at least one historical reconciliation result, wherein each historical reconciliation result is obtained by comparing the data of the first data source with the data of the second data source in at least one historical data partition before the current data partition; The cross-partition comparison module is used to compare the current reconciliation result with at least one of the historical reconciliation results, and determine the final reconciliation result of the current data partition based on the first comparison result.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the cross-partition reconciliation method as described in any one of claims 1 to 7.

10. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions that, when executed by the one or more processors, perform the steps of the cross-partition reconciliation method as described in any one of claims 1 to 7.

Images