A sliding window open bank data incremental synchronization cockpit method and device

Abstract

The application provides a sliding window open bank data incremental synchronization cockpit method and device, and the application comprises the following steps: obtaining a current synchronization water level line of a target data source, combining system time to construct a sliding query window containing a pre-overlapping time period; initiating a window incremental query request to a bank server through a program interface, obtaining newly added and changed original transaction data in the window; removing and standardizing the original transaction data to generate standard data events, injecting the standard data events into a message queue for stream aggregation calculation and updating a real-time materialized view, synchronously updating and persisting the synchronization water level line; dynamically determining a next synchronization time according to a server notification, a periodic scheduling and a system load, and pushing a data change amount to a front end based on the materialized view through a long connection to drive interface updating. The application realizes efficient synchronization, accurate deduplication and front-end visual real-time refreshing of bank transaction data increment, and guarantees data synchronization integrity and timeliness.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a sliding window method and apparatus for incremental synchronization of open banking data. Background Technology

[0002] As a core hub of the fintech ecosystem, open banking, by providing standardized APIs to third parties with access to account and transaction data, has been widely applied in real-time risk monitoring and operational decision-making. Among related technologies, data dashboard systems typically construct a complete process from source data retrieval to front-end visualization through a collaborative workflow of full polling, periodic scheduling, and local data cleansing. Specifically, this system encompasses key aspects such as scheduled task triggering, massive data acquisition, duplicate record deduplication, and indicator aggregation calculation, aiming to achieve dynamic awareness of business status.

[0003] However, existing synchronization methods directly employ a fixed-period full data retrieval mechanism without distinguishing between synchronized data and newly added / changed data. This can lead to massive amounts of invalid data transmission, causing API rate limiting or even service unavailability, or the lack of overlapping window design may result in the omission of delayed records arriving within the bank, thus affecting the integrity and real-time performance of the dashboard data. Furthermore, this rigid scheduling strategy cannot be dynamically adjusted according to the source load, resulting in a serious waste of computing resources and difficulty in guaranteeing end-to-end data consistency. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the related art.

[0005] Therefore, the first objective of this invention is to propose a sliding window method for incremental synchronization of open banking data.

[0006] Another objective of this invention is to provide a sliding window open banking data incremental synchronization cockpit device.

[0007] The third objective of this invention is to provide a computer device.

[0008] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0009] To achieve the above objectives, a first aspect of the present invention proposes a sliding window method for incremental synchronization of open banking data, comprising:

[0010] S10, obtain the current synchronization water level for the target data source, and construct a sliding query window containing the preceding overlapping time period based on the current synchronization water level and the current system time, wherein the preceding overlapping time period starts at a preset offset time before the current synchronization water level. S20, initiate an incremental data query request for the sliding query window to the bank server through the program interface, and receive and obtain the new or changed original transaction data generated in the sliding query window; S30, the acquired original transaction data is deduplicated and standardized to generate standard data events, the standard data events are injected into the message queue for real-time streaming aggregation calculation to update the real-time materialized view, and the current synchronization water level is updated to the end time of the sliding query window and then persistently stored. S40 dynamically determines the timing of the next synchronization trigger based on the notification signals from the bank server, the preset periodic scheduling strategy, and the system load status. Based on the updated real-time materialized view, it pushes the data change to the front end through a long connection channel to drive the update of the visualization interface.

[0011] In one embodiment of the present invention, S10 includes: Establish an independent synchronization session for each open banking data source to be synchronized, and read the last persistent storage water level value from a reliable storage medium or use the system's first startup time as the initial value to determine the current synchronization water level T_last; Get the current system time Now, and dynamically adjust the preset offset Δt based on the historical delay statistics of the bank system, where the value of Δt ranges from 5 seconds to 60 seconds; Based on the current synchronization water level T_last, the preset offset Δt, and the current system time Now, a query time window of [T_last-Δt,Now] is constructed to actively capture records that are delayed due to internal processing delays in the bank system.

[0012] In one embodiment of the present invention, the step of dynamically adjusting the preset offset Δt based on historical delay statistics of the banking system includes: Real-time statistics are collected on the distribution of data write delays to the bank's server within a preset time period, and the maximum time difference in the arrival of records is calculated. Multiplying the maximum time difference by a preset safety factor yields a dynamically adjusted candidate offset value. If the dynamically adjusted offset candidate value is less than the minimum allowed overlap time, the preset offset Δt is set to the minimum allowed overlap time; otherwise, the preset offset Δt is set to the dynamically adjusted offset candidate value to ensure that the sliding query window can cover the vast majority of delayed transaction data.

[0013] In one embodiment of the present invention, S20 includes: Based on the open banking API interface specification corresponding to the target data source, construct an HTTP request message containing the start and end times of the sliding query window; The system sends HTTP request messages to the bank's server in batches through a standardized API interface and receives raw transaction data response packets in JSON or XML format returned by the bank's server. The original transaction data response package is initially verified, including data format integrity verification and timestamp validity verification. Only the original transaction data that passes the verification is retained as the basic data stream for subsequent processing.

[0014] In one embodiment of the present invention, S30 includes: Extract the business primary key K from each piece of data in the raw transaction data event stream D_raw injected into the unified message queue, where K is equal to the concatenation hash value of the transaction serial number transaction_id and the data source API identifier source_api_id; First, the business primary key K is initially screened using a Bloom filter BF. If BF.contains(K) returns true, it is then entered into the exact hash table H for a second verification. If the business primary key K does not belong to the exact hash table H, it is determined to be valid new data, and the operation of H equals the union of H and {K} is performed and the Bloom filter BF is updated; otherwise, the duplicate event is discarded. The deduplicated data is standardized and converted according to the data structure definition of the UnifiedTransactionEvent entity class to generate a standard data event containing the transaction serial number, transaction timestamp, unique account identifier, transaction amount, transaction currency, transaction type, transaction status, counterparty account identifier, transaction location code, affiliated branch code, financial product type, customer level, risk weight coefficient, data source API identifier, data version number, and end-to-end tracking identifier.

[0015] In one embodiment of the present invention, S40 includes: The front-end and back-end services of the cockpit establish a long WebSocket connection channel, and the front-end chart component subscribes to specific data stream topics or aggregate result topics from the back-end according to its data needs. The backend service listens for change events of the real-time materialized view, and after detecting an update, extracts the difference in the changed data or the increment of the new aggregation result; The WebSocket long connection channel is used to push only the changed data difference or the new aggregation result increment to the front end; After receiving incremental data, the front end does not need to refresh the entire page. Instead, it performs smooth animation updates only on specific chart components that have subscribed to the corresponding theme, so as to achieve millisecond-level data visualization.

[0016] In one embodiment of the present invention, it further includes: Generate a unique tracking ID for each batch of synchronized data and record full-link logs, and periodically start offline integrity verification jobs; The process of generating a unique tracking ID for each batch of synchronized data and recording the entire link log includes: embedding log collection probes in each link from API calls, message queue delivery, stream processing to cockpit rendering, and recording the unique tracking ID throughout the entire link log to achieve traceability of the data flow process; The periodic offline integrity verification operation includes: at a preset verification time point, comparing the incrementally accumulated data with the full snapshot data provided by the bank API; if a difference is found, an automatic compensation synchronization mechanism is triggered, and a difference report containing details of the difference is generated to ensure the final consistency between the dashboard data and the bank's source data.

[0017] To achieve the above objectives, a second aspect of the present invention provides a sliding window open banking data incremental synchronization cockpit device, comprising: A window building module is used to obtain the current synchronization water level for the target data source, and build a sliding query window containing a preceding overlapping time period based on the current synchronization water level and the current system time, wherein the preceding overlapping time period starts at a preset offset time before the current synchronization water level. The incremental query module is used to initiate an incremental data query request for the sliding query window to the bank server through the program interface, and to receive and obtain the new or changed original transaction data generated in the sliding query window. The data processing and aggregation module is used to perform deduplication and standardization transformation on the acquired raw transaction data to generate standard data events, inject the standard data events into the message queue for real-time streaming aggregation calculation to update the real-time materialized view, and persist the current synchronization water level to the end time of the sliding query window. The synchronization scheduling and push module is used to dynamically determine the timing of the next synchronization trigger based on the notification signals from the bank server, the preset periodic scheduling strategy, and the system load status. Based on the updated real-time materialized view, it pushes the data change to the front end through a long connection channel to drive the update of the visualization interface.

[0018] This invention discloses a sliding window-based open banking data incremental synchronization dashboard method and apparatus. Through sliding window incremental synchronization and a pre-overlapping mechanism, it significantly reduces network transmission overhead and resolves data loss issues caused by data write latency, ensuring data integrity. Combined with adaptive scheduling and real-time stream processing, it reduces end-to-end data latency from minutes to milliseconds, effectively alleviating pressure on API servers and improving the dashboard's real-time response capabilities.

[0019] To achieve the above objectives, a third aspect of this application provides a computer device, including a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading executable program code stored in the memory, for implementing the method described in the first aspect embodiment.

[0020] To achieve the above objectives, a fourth aspect of this application provides a non-transitory computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect.

[0021] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0022] Figure 1 This is a flowchart of a sliding window open banking data incremental synchronization cockpit method according to an embodiment of the present invention; Figure 2 This is a structural diagram of a sliding window open banking data incremental synchronization cockpit device according to an embodiment of the present invention; Figure 3 It is a computer device according to an embodiment of the present invention. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0025] The following description, with reference to the accompanying drawings, describes a sliding window-based open banking data incremental synchronization cockpit method and apparatus according to an embodiment of the present invention.

[0026] Figure 1 This is a flowchart of a sliding window open banking data incremental synchronization cockpit method according to an embodiment of the present invention, as shown below. Figure 1 As shown, it includes: S10, obtain the current synchronization water level for the target data source, and construct a sliding query window containing the preceding overlapping time period based on the current synchronization water level and the current system time, wherein the preceding overlapping time period starts at a preset offset time before the current synchronization water level. S20, initiate an incremental data query request for the sliding query window to the bank server through the program interface, and receive and obtain the new or changed original transaction data generated in the sliding query window; S30, the acquired original transaction data is deduplicated and standardized to generate standard data events, the standard data events are injected into the message queue for real-time streaming aggregation calculation to update the real-time materialized view, and the current synchronization water level is updated to the end time of the sliding query window and then persistently stored. S40 dynamically determines the timing of the next synchronization trigger based on the notification signals from the bank server, the preset periodic scheduling strategy, and the system load status. Based on the updated real-time materialized view, it pushes the data change to the front end through a long connection channel to drive the update of the visualization interface.

[0027] This invention proposes another sliding window method for incremental synchronization of open banking data in a cockpit, which includes the following steps: Step S1: Initialize the synchronization water level. Establish an independent synchronization session for each open banking data source to be synchronized. Initialize the synchronization water level T_last to the system's first startup time or the water level value of the last persistent storage.

[0028] Step S2: Perform a sliding window incremental query. Based on the current water level T_last and the current system time Now, construct a query time window [T_last-Δt,Now], where Δt is a preset preceding overlap time offset (e.g., 5 to 60 seconds, which can be dynamically adjusted according to the historical delay statistics of the bank system). Initiate an incremental data query request within this time window through the Open Banking API.

[0029] Step S3: Process the query results and update the water level. Receive incremental data returned by the API and perform preliminary data validation (such as data format validation and timestamp validity validation). Encapsulate the validated data into data events and inject them into a unified message queue. Update the water level T_last to the end time Now of the current query window and persist the updated water level to a reliable storage medium (such as a database or distributed key-value store).

[0030] Step S4: Adaptively schedule the next synchronization, determining the trigger timing for the next synchronization based on the current system status: If a Webhook notification is received from the bank, the next synchronization is triggered immediately; if no Webhook notification is received, it is triggered after a preset periodic scheduling time (e.g., 30 seconds); after each synchronization is completed, the API response time, its own processing queue length, and the bank's load indicators are evaluated; if the API response time exceeds a preset threshold or the processing queue length exceeds a safe level, the synchronization cycle is automatically extended; if it is during peak business hours and the system load is normal, the synchronization interval is intelligently shortened.

[0031] Step S5: Stream processing and real-time aggregation. The stream processing engine subscribes to data events in the message queue and performs the following steps in sequence: Deduplication: Based on the business primary key (such as transaction serial number), duplicate data caused by sliding window overlap or network retries is eliminated. Format standardization: The data formats of different bank APIs are uniformly converted into the system's internal standard data model; the system's internal standard data model is a unified transaction event entity class, whose data structure includes the following fields: Transaction ID: transaction_id, data type string, serves as a globally unique business primary key to uniquely identify a single transaction record; Transaction timestamp: The data type is long integer and the unit is milliseconds. It is used to record the actual time when the transaction occurred. Unique account identifier: account_id, data type string, used to uniquely identify the account to which a transaction belongs; Transaction amount: amount, data type is high precision numeric type, accurate to two decimal places, used to represent the amount of funds in the transaction; Transaction currency: currency, data type is string, follows ISO4217 international standard encoding, used to standardize the identification of transaction currency; Transaction type: transaction_type, data type is string, is a preset enumeration type, including expenditure, deposit, transfer, and reversal; Transaction status: status, data type is string, is a preset enumeration type, including success, failure, processing, timeout; Counterparty account identifier: counterparty_id, a string type, used to identify the counterparty account information in a transaction; Transaction location code: region_code, a string data type, used to represent the administrative region where the transaction occurred; Branch ID: branch_id, a string representing the bank branch to which the transaction belongs; Financial product type: product_type, a string representing the type of financial product corresponding to the transaction; Customer level: customer_level, a string data type, is used to identify the level attribute of the customer to whom the transaction belongs; Risk weight coefficient: risk_weight, data type is floating point, value range is [0.0, 1.0], used to represent the risk level of the transaction; Data source API identifier: source_api_id, data type is string, used to uniquely identify the source interface of transaction data; Data version number: version, data type is integer, used to implement optimistic locking control of data and ensure data update consistency; End-to-end tracing identifier: trace_id, data type is string, UUID format, used to realize end-to-end tracing and traceability of transaction data.

[0032] Dimensional Association: Associate business dimension information, such as affiliated branch, product type, customer level, etc.; Real-time Aggregation Calculation: Calculate key business indicators in real time on the data stream, such as total transactions per second (TPS), transaction distribution by region, real-time success rate, real-time risk score, etc.; Continuously update the calculation results to the cache to form a real-time materialized view.

[0033] Step S6: Real-time rendering and push of the cockpit. The cockpit front-end and back-end services establish a WebSocket long connection; the front-end chart component subscribes to specific data stream topics or aggregation result topics from the back-end according to its data requirements; the back-end service listens for change events of the real-time materialized view, and after detecting an update, only pushes the changed data difference or new aggregation result incrementally to the front-end through the WebSocket channel; after receiving the incremental data, the front-end does not need to refresh the entire page, but only performs smooth animation updates on specific chart components.

[0034] Step S7: End-to-end monitoring and consistency verification. Generate a unique tracking ID for each batch of synchronized data and record the entire process log from API calls, message queue delivery, stream processing to cockpit rendering. Start an offline integrity verification job regularly (e.g., every early morning) to compare the cumulative data after incremental accumulation with the full snapshot provided by the bank API. If a difference is found, automatically trigger the compensation synchronization mechanism and generate a difference report.

[0035] Furthermore, the present invention provides an open banking real-time data cockpit system based on time window sliding incremental synchronization, the system comprising: The incremental synchronization engine module maintains a dynamic synchronization watermark T_last for each data source, representing the time point of the last successful data acquisition. During each synchronization, the incremental synchronization engine initiates a sliding window query with a time range of [T_last-Δt, Now], where Δt is a preset preceding overlap time offset, used to proactively capture records delayed due to internal processing delays within the bank system. After successfully processing data in a window, the watermark T_last is updated to the end time of the current window and persistently stored. The adaptive scheduling module dynamically adjusts the synchronization triggering strategy based on data characteristics and system load. The strategies include: periodic scheduling triggering based on a preset period, event-driven triggering based on bank-side proactive notifications (Webhook), and adaptive speed adjustment triggering based on API response time and its own processing queue length. The stream processing pipeline module injects incremental data as continuous data events into a unified message queue and performs deduplication, format standardization, standardization and enrichment processing of related business dimensions, as well as real-time calculation of key business indicators on the data stream to form a real-time materialized view. The deduplication process employs a two-stage deduplication mechanism: a Bloom filter based on the business primary key and a precise hash table. Let the input data event stream be D = {d1, d2, ..., d...}. n For each data entry, extract the business primary key K = Hash(transaction_id||source_api_id). First, perform an initial screening using a Bloom filter (BF). If BF.contains(K) = true, then proceed to the precise hash table H for a second verification; otherwise, K... If H is found to be valid new data, H = H∪{K} is executed and BF is updated; otherwise, the duplicate event is discarded. The real-time aggregation calculation is performed based on a sliding time window. Let the current time be t, and the window size be T_w, then the current window W(t) = {d i |t-T_w≤d i The key indicator is calculated using the formula: .timestamp≤t}. Transactions per second (TPS(t)) = |{d i |d i ∈W(t)∧d i .status='SUCCESS'}| / T_w; Total transaction amount GMV(t) = Σd i .amount, where d i ∈W(t)∧d i .status='SUCCESS'; Real-time success rate SuccessRate(t) = |{d i |d i ∈W(t)∧d i .status='SUCCESS'}| / |W(t)|×100%; Regional distribution ratio Dist(r,t)=|{d i |d i ∈W(t)∧d i .region_code=r}| / |W(t)|; Risk score (t) = Σ(d) i .amount×d i .risk_weight) / GMV(t), where risk_weight is the preset risk weight coefficient, which is 0 when GMV(t)=0; The input-output process is as follows: The input is the raw incremental data event stream D_raw from the message queue. After deduplication, standardization, and dimension correlation, it forms a standardized event stream D_std. Then, after real-time aggregation and calculation, the output is an indicator update stream ΔM, which is finally written to the cache to form a real-time materialized view V_realtime, satisfying the following: V_realtime(t)=V_realtime(t-1)+ΔM(t); Furthermore, the cockpit rendering push module is used to establish communication with the front end via a long connection, subscribe to specific data streams or aggregation result topics according to the data requirements of the front end chart components, and push only the changed data differences or new aggregation result increments to the front end after the real-time materialized view is updated, so that the front end can perform partial updates to specific charts; the monitoring and consistency assurance module is used to generate a unique tracking ID for each batch of synchronized data and record full-link logs, as well as periodically start offline integrity verification jobs to compare the incrementally accumulated data with the full snapshot provided by the bank API.

[0036] The embodiments of this invention also have the following technical effects: Significantly reducing system load and operating costs. This invention uses an incremental synchronization mechanism to replace full polling, only fetching newly added or changed data within the time window. Actual calculations show that in typical open banking scenarios (daily transaction volume in the tens of millions, data change rate approximately 5%-10%), invalid data transmission volume and API call count can be reduced by more than 20%. Simultaneously, since there is no need to repeatedly process unchanged historical data, the CPU and memory resource consumption of the data processing pipeline is also significantly reduced, significantly alleviating the pressure on the bank's API server and its own infrastructure. Achieving true near real-time data visualization, this invention uses a trigger mechanism that integrates event-driven and periodic scheduling, combined with incremental push technology using WebSocket long connections, to reduce the end-to-end latency from hours / minutes in traditional solutions to seconds or even milliseconds from the data generation at the bank's source to the dashboard presentation. Business personnel can gain near-instantaneous insights into transaction trends, risk conditions, and business indicators, providing strong support for real-time decision-making. To fundamentally ensure data integrity and consistency, the "pre-overlapping window" design ([T_last-Δt,Now]) proactively covers the data processing delay window within the bank system, ensuring that delayed records are reliably captured, fundamentally solving the data loss problem caused by fixed-time point queries. The waterline persistence mechanism enables precise breakpoint resume functionality, allowing synchronization to continue from the last interrupted position even after a restart of the synchronization process or a brief service interruption, ensuring data continuity. Furthermore, regular offline integrity verification operations form a dual guarantee of "real-time incremental + offline verification," ensuring the final consistency between the dashboard data and the bank's source data. Demonstrating friendliness to API service providers and system robustness, the adaptive speed adjustment strategy dynamically adjusts the synchronization frequency based on the actual load on the bank's end, proactively yielding when the bank's load is high and intelligently accelerating when business needs dictate, reflecting friendliness to open banking API service providers and reducing the risk of rate limiting or service degradation due to high-frequency calls. The end-to-end tracing mechanism provides complete data support for problem localization, performance optimization, and compliance auditing. This invention provides a high-quality real-time data foundation for upper-layer intelligent applications. The standardized and timely data streams produced by this invention can directly serve advanced financial technology applications such as real-time risk warning, dynamic credit limit control, and instant marketing feedback, without the need for additional data extraction and transformation steps, thus improving the development efficiency and data quality of upper-layer applications.

[0037] To achieve the above embodiments, such as Figure 2 As shown, this embodiment also provides a sliding window open banking data incremental synchronization cockpit device 10, including: The window construction module 100 is used to obtain the current synchronization water level for the target data source, and construct a sliding query window containing a preceding overlapping time period based on the current synchronization water level and the current system time, wherein the preceding overlapping time period starts at a preset offset time before the current synchronization water level. The incremental query module 200 is used to initiate an incremental data query request for the sliding query window to the bank server through a program interface, and to receive and obtain the new or changed original transaction data generated in the sliding query window. The data processing and aggregation module 300 is used to perform deduplication and standardization transformation on the acquired raw transaction data to generate standard data events, inject the standard data events into the message queue for real-time streaming aggregation calculation to update the real-time materialized view, and update the current synchronization water level to the end time of the sliding query window and then persist the data. The synchronization scheduling and push module 400 is used to dynamically determine the timing of the next synchronization trigger based on the notification signals from the bank server, the preset periodic scheduling strategy, and the system load status, and push the data change amount to the front end through a long connection channel based on the updated real-time materialized view to drive the update of the visualization interface.

[0038] This invention discloses a sliding window-based open banking data incremental synchronization dashboard device. Through sliding window incremental synchronization and a pre-overlapping mechanism, it significantly reduces network transmission overhead and resolves data loss issues caused by data write latency, ensuring data integrity. Combined with adaptive scheduling and real-time stream processing, it reduces end-to-end data latency from minutes to milliseconds, effectively alleviating pressure on API servers and improving the dashboard's real-time response capabilities.

[0039] To implement the methods of the above embodiments, the present invention also provides a computer device, such as... Figure 3 As shown, the computer device 600 includes a memory 601 and a processor 602; wherein, the processor 602 reads executable program code stored in the memory 601 to run a program corresponding to the executable program code, so as to implement the various steps of the method described above.

[0040] To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0041] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0042] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A sliding window-based open banking data incremental synchronization cockpit method, characterized in that, include: S10, obtain the current synchronization water level for the target data source, and construct a sliding query window containing the preceding overlapping time period based on the current synchronization water level and the current system time, wherein the preceding overlapping time period starts at a preset offset time before the current synchronization water level. S20, initiate an incremental data query request for the sliding query window to the bank server through the program interface, and receive and obtain the new or changed original transaction data generated in the sliding query window; S30, the acquired original transaction data is deduplicated and standardized to generate standard data events, the standard data events are injected into the message queue for real-time streaming aggregation calculation to update the real-time materialized view, and the current synchronization water level is updated to the end time of the sliding query window and then persistently stored. S40 dynamically determines the timing of the next synchronization trigger based on the notification signals from the bank server, the preset periodic scheduling strategy, and the system load status. Based on the updated real-time materialized view, it pushes the data change to the front end through a long connection channel to drive the update of the visualization interface.

2. The method as described in claim 1, characterized in that, S10 includes: Establish an independent synchronization session for each open banking data source to be synchronized, and read the last persistent storage water level value from a reliable storage medium or use the system's first startup time as the initial value to determine the current synchronization water level T_last; Get the current system time Now, and dynamically adjust the preset offset Δt based on the historical delay statistics of the bank system, where the value of Δt ranges from 5 seconds to 60 seconds; Based on the current synchronization water level T_last, the preset offset Δt, and the current system time Now, a query time window of [T_last-Δt,Now] is constructed to actively capture records that are delayed due to internal processing delays in the bank system.

3. The method as described in claim 2, characterized in that, The step of dynamically adjusting the preset offset Δt based on historical delay statistics from the banking system includes: Real-time statistics are collected on the distribution of data write delays to the bank's server within a preset time period, and the maximum time difference in arrival of records is calculated. Multiplying the maximum time difference by a preset safety factor yields a dynamically adjusted candidate offset value. If the dynamically adjusted offset candidate value is less than the minimum allowed overlap time, the preset offset Δt is set to the minimum allowed overlap time; otherwise, the preset offset Δt is set to the dynamically adjusted offset candidate value to ensure that the sliding query window can cover the vast majority of delayed transaction data.

4. The method as described in claim 1, characterized in that, S20 includes: Based on the open banking API interface specification corresponding to the target data source, construct an HTTP request message containing the start and end times of the sliding query window; The system sends HTTP request messages to the bank's server in batches through a standardized API interface and receives raw transaction data response packets in JSON or XML format returned by the bank's server. The original transaction data response package is initially verified, including data format integrity verification and timestamp validity verification. Only the original transaction data that passes the verification is retained as the basic data stream for subsequent processing.

5. The method as described in claim 1, characterized in that, S30 includes: Extract the business primary key K from each piece of data in the raw transaction data event stream D_raw injected into the unified message queue, where K is equal to the concatenation hash value of the transaction serial number transaction_id and the data source API identifier source_api_id; First, the business primary key K is initially screened using a Bloom filter BF. If BF.contains(K) returns true, it is then entered into the exact hash table H for a second verification. If the business primary key K does not belong to the exact hash table H, it is determined to be valid new data, and the operation of H equals the union of H and {K} is performed and the Bloom filter BF is updated; otherwise, the duplicate event is discarded. The deduplicated data is standardized and converted according to the data structure definition of the UnifiedTransactionEvent entity class to generate a standard data event containing the transaction serial number, transaction timestamp, unique account identifier, transaction amount, transaction currency, transaction type, transaction status, counterparty account identifier, transaction location code, affiliated branch code, financial product type, customer level, risk weight coefficient, data source API identifier, data version number, and end-to-end tracking identifier.

6. The method as described in claim 1, characterized in that, S40 includes: The front-end and back-end services of the cockpit establish a long WebSocket connection channel, and the front-end chart component subscribes to specific data stream topics or aggregate result topics from the back-end according to its data needs. The backend service listens for change events of the real-time materialized view, and after detecting an update, extracts the difference in the changed data or the increment of the new aggregation result; The WebSocket long connection channel is used to push only the changed data difference or the new aggregation result increment to the front end; After receiving incremental data, the front end does not need to refresh the entire page. Instead, it performs smooth animation updates only on specific chart components that have subscribed to the corresponding theme, so as to achieve millisecond-level data visualization.

7. The method as described in claim 1, characterized in that, The method also includes: Generate a unique tracking ID for each batch of synchronized data and record full-link logs, and periodically start offline integrity verification jobs; The process of generating a unique tracking ID for each batch of synchronized data and recording the entire link log includes: embedding log collection probes in each link from API calls, message queue delivery, stream processing to cockpit rendering, and recording the unique tracking ID throughout the entire link log to achieve traceability of the data flow process; The periodic offline integrity verification operation includes: at a preset verification time point, comparing the incrementally accumulated data with the full snapshot data provided by the bank API; if a difference is found, an automatic compensation synchronization mechanism is triggered, and a difference report containing details of the difference is generated to ensure the final consistency between the dashboard data and the bank's source data.

8. A sliding window-based open banking data incremental synchronization cockpit device, characterized in that, include: A window building module is used to obtain the current synchronization water level for the target data source, and build a sliding query window containing a preceding overlapping time period based on the current synchronization water level and the current system time, wherein the preceding overlapping time period starts at a preset offset time before the current synchronization water level. The incremental query module is used to initiate an incremental data query request for the sliding query window to the bank server through the program interface, and to receive and obtain the new or changed original transaction data generated in the sliding query window. The data processing and aggregation module is used to perform deduplication and standardization transformation on the acquired raw transaction data to generate standard data events, inject the standard data events into the message queue for real-time streaming aggregation calculation to update the real-time materialized view, and persist the current synchronization water level to the end time of the sliding query window. The synchronization scheduling and push module is used to dynamically determine the timing of the next synchronization trigger based on the notification signals from the bank server, the preset periodic scheduling strategy, and the system load status. Based on the updated real-time materialized view, it pushes the data change to the front end through a long connection channel to drive the update of the visualization interface.

9. A computer device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement a sliding window open banking data incremental synchronization cockpit method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a sliding window open banking data incremental synchronization cockpit method as described in any one of claims 1-7.

Images