Lake and warehouse integrated financial level-based multi-source heterogeneous data synchronization method, device and equipment and storage medium

Abstract

The invention discloses a lake and warehouse integrated financial level-based multi-source heterogeneous data synchronization method, device and equipment and a storage medium, and relates to the technical field of data processing, the method comprises the following steps: in response to a data synchronization request, performing data fragmentation on to-be-synchronized total data based on the data synchronization request to obtain total data blocks; the full-amount data blocks are distributed to a full-amount data synchronization concurrent thread in a polling mode for full-amount data synchronization; under the condition that synchronization of the to-be-synchronized full data is completed, data fragmentation is conducted on the to-be-synchronized incremental data based on a lock-free consistency switching strategy, incremental data synchronization is conducted on the basis of the obtained target data block, and the lock-free consistency switching strategy is a snapshot and incremental data cooperative processing strategy based on a watermark mechanism. Compared with an existing mode that read lock processing needs to be conducted on the read library or table during data synchronization, data synchronization can be conducted under lock-free operation in the mode, and business transaction is prevented from being blocked.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, equipment and storage medium for synchronizing financial-grade multi-source heterogeneous data based on lake warehouse integration. Background Technology

[0002] A financial enterprise-grade data warehouse refers to a large-scale, highly reliable, secure, and high-performance data storage and analysis platform specifically built for financial institutions (such as banks, insurance companies, and consumer finance companies). It is primarily used to integrate structured data from different business systems (such as core banking systems, credit systems, risk control systems, and payment systems) to support critical business decisions such as operational analysis, risk control, customer management, compliance supervision, and anti-fraud. Currently, financial-grade data warehouses face numerous challenges, including a predominance of structured data, insufficient real-time performance, a conflict between consistency guarantees and performance, and difficulties in integrating multi-source data. Therefore, the industry is gradually evolving towards integrated lake warehouses, integrated stream and batch processing warehouses, and real-time data warehouses to meet the increasingly complex needs of financial businesses. Financial-grade lake warehouse platforms have extremely high requirements for "consistency and accuracy." Currently, most integrated lake warehouse synchronization technologies require read locks on the databases or tables being read to ensure consistency. Global locks can cause the database to fail to respond to user requests, resulting in a database hang. This problem is critical in financial scenarios and can impact online business operations. Summary of the Invention

[0003] The main purpose of this application is to provide a method, apparatus, device and storage medium for financial-grade multi-source heterogeneous data synchronization based on lake warehouse integration, which aims to solve the technical problem that the existing data synchronization through locking methods affects business processes.

[0004] To achieve the above objectives, this application proposes a lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method, which includes: In response to a data synchronization request, the full data to be synchronized is fragmented based on the data synchronization request to obtain a full data block; The full data blocks are distributed in a round-robin fashion to the full data synchronization concurrent thread for full data synchronization; Once the full data to be synchronized is synchronized, the incremental data to be synchronized is sharded based on a lock-free consistency switching strategy, and incremental data synchronization is performed based on the obtained target data blocks. The lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on a watermarking mechanism.

[0005] Optionally, the step of performing data fragmentation on the incremental data to be synchronized based on the lock-free consistency switching strategy, and performing incremental data synchronization based on the obtained target data blocks, includes: Pause the processing of change logs and perform data sharding on the incremental data to be synchronized to obtain initial data blocks; The initial data block is deduplicated to obtain the target data block; Incremental data synchronization is performed based on the target data block.

[0006] Optionally, the step of deduplicating the initial data block to obtain the target data block includes: Obtain high-watermark markers and restore the change log processing flow; The initial data block is deduplicated based on the paused processing node, the low watermark, and the high watermark marker to obtain the target data block, wherein the paused processing node is the processing node when the change log is paused.

[0007] Optionally, after the step of responding to the data synchronization request, the method further includes: The database export tool is used to synchronize all the data to be synchronized to the data warehouse tables. Write the full amount of data to be synchronized into the financial-grade lake warehouse platform according to the data warehouse table; Send the incremental data to be synchronized to the message queue; The incremental data to be synchronized is written to the financial-grade lake warehouse platform based on the message queue.

[0008] Optionally, the target of the data synchronization is a financial-grade lake warehouse platform; After the steps of performing data sharding on the incremental data to be synchronized based on the lock-free consistency switching strategy and synchronizing the incremental data based on the sharding results, when the full data to be synchronized is completed, the method further includes: Obtain the semi-structured and unstructured data to be synchronized; The semi-structured data and the unstructured data are synchronized to the financial-grade lake warehouse platform.

[0009] Optionally, the step of synchronizing the semi-structured data to the financial-grade lakeware platform includes: The semi-structured data is stored in a preset file system, and the file path corresponding to the semi-structured data is recorded. The semi-structured data is parsed based on the file path to obtain structured data; The structured data is synchronized to the financial-grade lake warehouse platform.

[0010] Optionally, the step of synchronizing the unstructured data to the financial-grade lakeware platform includes: The unstructured data is stored in a preset file system, and the file path corresponding to the unstructured data is recorded. The unstructured data is vectorized based on the file path to obtain vectorized data; The vectorized data is processed according to a preset data processing model to obtain the model processing result; The model processing results are synchronized to the financial-grade lake warehouse platform.

[0011] Furthermore, to achieve the above objectives, this application also proposes a lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization device, which includes: The response module is used to respond to a data synchronization request and, based on the data synchronization request, perform data fragmentation on the full amount of data to be synchronized to obtain a full data block; The full data synchronization module is used to distribute the full data blocks in a round-robin fashion to the full data synchronization concurrent thread for full data synchronization; The acquisition module is used to acquire incremental data to be synchronized through a lock-free consistency switching strategy after the full data to be synchronized has been synchronized. The incremental data synchronization module is used to perform data fragmentation on the incremental data to be synchronized based on a lock-free consistency switching strategy after the full data to be synchronized has been synchronized. The incremental data synchronization is performed based on the obtained target data block. The lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on a watermarking mechanism.

[0012] Furthermore, to achieve the above objectives, this application also proposes a lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method described above.

[0013] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method described above.

[0014] In response to a data synchronization request, this application shards the full data to be synchronized based on the request to obtain full data blocks; the full data blocks are then round-robin distributed to concurrent threads for full data synchronization; once the full data synchronization is complete, the incremental data to be synchronized is sharded based on a lock-free consistency switching strategy, and incremental data synchronization is performed based on the obtained target data blocks. The lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on a watermarking mechanism. Compared to existing methods that require read locks on the database or table being read during data synchronization, the method described in this application enables data synchronization without locks, preventing business transactions from being blocked. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a flowchart illustrating an embodiment of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method provided in this application. Figure 2 This is a schematic diagram of the watermark-based block selection process provided in Embodiment 1 of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method of this application; Figure 3 This is a schematic diagram of the overall data synchronization process provided in Embodiment 1 of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method of this application; Figure 4 This is a flowchart illustrating Embodiment 2 of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method provided in this application; Figure 5 This is a schematic diagram of the module structure of the lake warehouse integrated financial-grade multi-source heterogeneous data synchronization device in an embodiment of this application; Figure 6 This is a schematic diagram of the hardware operating environment involved in the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method in the embodiments of this application.

[0018] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0019] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0020] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0021] The main solution of this application embodiment is as follows: In response to a data synchronization request, data sharding is performed on the full data to be synchronized based on the data synchronization request to obtain full data blocks; the full data blocks are then round-robin allocated to concurrent threads for full data synchronization; when the full data to be synchronized is completed, data sharding is performed on the incremental data to be synchronized based on a lock-free consistency switching strategy, and incremental data synchronization is performed based on the obtained target data blocks. The lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on a watermarking mechanism. Compared with existing methods that require read locks on the database or table being read during data synchronization, the above method of this application can perform data synchronization without locks, avoiding blocking of business transactions.

[0022] It should be noted that the executing entity in this embodiment can be a computing service device with functions such as lakeware-integrated financial-grade multi-source heterogeneous data synchronization, network communication, and program execution, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of realizing the above functions, or a lakeware-integrated financial-grade multi-source heterogeneous data synchronization device. The following description uses a lakeware-integrated financial-grade multi-source heterogeneous data synchronization device as an example to illustrate this embodiment and the subsequent embodiments.

[0023] Based on this, embodiments of this application provide a method for synchronizing multi-source heterogeneous data at a financial grade based on a lake warehouse, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method of this application.

[0024] In this embodiment, the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method includes the following steps: Step S10: In response to the data synchronization request, the full data to be synchronized is fragmented based on the data synchronization request to obtain a full data block; It should be noted that the data synchronization request can be a data synchronization command that synchronizes data from different business systems (such as banking systems, credit systems, risk control systems, payment systems, etc.) to a financial-grade lakeware platform integrating lakeware. The full data to be synchronized in this step is structured data. The step of data sharding based on the data synchronization request to obtain full data blocks can be achieved by sharding the full data to be synchronized using methods such as primary key range, fixed-quantity equal distribution, or database partitioning, resulting in sharded full data blocks.

[0025] Step S20: The full data block is distributed in a round-robin fashion to the full data synchronization concurrent thread for full data synchronization; It should be noted that the process of allocating the full data blocks to the full data synchronization concurrent thread for full data synchronization can involve creating multiple concurrent threads simultaneously for full data synchronization, and then allocating the full data blocks obtained from data sharding to the full data synchronization concurrent thread in a round-robin fashion, thereby performing full data synchronization through the full data synchronization concurrent thread.

[0026] Step S30: After the full data to be synchronized is synchronized, the incremental data to be synchronized is sharded based on the lock-free consistency switching strategy, and incremental data is synchronized based on the obtained target data block. The lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on the watermarking mechanism.

[0027] It should be understood that the watermark in a watermarking mechanism is a logical timestamp or location identifier, indicating that "all data before this time / location should have arrived in the system." It is used to handle issues related to event time, out-of-order data, window calculations, and incremental synchronization. The core function of watermarking is to define a "safe data range," allowing us to know which data is complete and safe to process.

[0028] It should be noted that the lock-free consistency switching strategy can be a watermark-based incremental synchronization process, the steps of which include: capturing incremental data streams: continuously acquiring changed data from the data source, each data item carrying an event time (event_time); generating watermarks: the system dynamically generates a watermark based on the event time, indicating that "data with event_time <= watermark should have all arrived"; defining watermark intervals: defining a time range based on the current watermark and the previous watermark, such as [previous watermark, current watermark), as the incremental data interval to be processed this time (this step is the logical basis for data sharding); selecting / extracting target data shards: based on the watermark interval, selecting data from the data source or buffer whose event_time falls within the interval, as the data shards (Chunks) to be synchronized / processed this time; deduplication / out-of-order processing: performing deduplication, sorting, window calculation, and other processing on the selected data shards to ensure data accuracy; synchronizing / writing to the target: writing the processed data shards (i.e., the target data blocks) to the financial-grade lakeware platform.

[0029] Furthermore, in order to perform lock-free operations during data synchronization and avoid database hangs, the steps of performing data sharding on the incremental data to be synchronized based on the lock-free consistency switching strategy, and performing incremental data synchronization based on the obtained target data blocks, include: Pause the processing of change logs and perform data sharding on the incremental data to be synchronized to obtain initial data blocks; The initial data block is deduplicated to obtain the target data block; Incremental data synchronization is performed based on the target data block.

[0030] It should be noted that the change logs mentioned include database binlogs and redologs. The pause on change logs can be implemented after full synchronization, by temporarily halting consumption of change logs (such as binlogs and redologs) to prevent newly arriving changes from interfering with subsequent data sharding and synchronization logic. The process of sharding the incremental data to be synchronized to obtain initial data blocks can be based on a certain logic (such as a pause point, low watermark, etc.), "splitting" a logical data range from the change logs or data stream as the initial incremental data set (i.e., the initial data block), which will be the object of subsequent processing. The initial data block may contain: data already synchronized during the full synchronization phase (duplicates), and data that has changed again in the change logs after the pause but before deduplication (redundancy or conflicts). Therefore, it is necessary to further filter / remove these duplicate or invalid data, retaining only the truly needed "net incremental data," i.e., the target data block. Then, incremental data synchronization is performed based on the target data block, that is, the target data block obtained after sharding and deduplication (i.e., the truly needed incremental data) is securely and efficiently synchronized to the financial-grade lakeware platform.

[0031] Furthermore, the step of deduplicating the initial data block to obtain the target data block includes: Obtain high-watermark markers and restore the change log processing flow; The initial data block is deduplicated based on the paused processing node, the low watermark, and the high watermark marker to obtain the target data block, wherein the paused processing node is the processing node when the change log is paused.

[0032] It should be noted that resuming change log processing involves restarting the listening and consumption of change logs after the initial data block selection is completed, in order to capture subsequent possible changes (for deduplication judgment). The pause node is the processing position where change log consumption was paused (e.g., Binlog Position, Kafka Offset, a specific record ID). The low watermark (L) is a starting point determined at or before the pause, indicating that "the data in the initial data block comes from changes before this point." The high watermark (H) is an ending point determined after log recovery, indicating that "we only care about changes between L and H." Deduplication of the initial data block based on the pause node, low watermark, and high watermark can be achieved by determining whether each piece of data in the initial data block has also appeared in the change log within the range [L, H) (possibly updated / rewritten / deleted). If so, it is considered duplicate or overwritten data and is removed; otherwise, it is retained, resulting in the target data block.

[0033] In specific implementation, please refer to Figure 2 , Figure 2 This is a schematic diagram of the watermark-based block selection process provided in Embodiment 1 of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method of this application; Figure 2 This demonstrates a watermark-based chunk selection process for accurately selecting data chunks in a change log. The core logic involves marking ranges with "low watermark (L)" and "high watermark (H)" and combining this with deduplication operations to ensure the integrity and uniqueness of data chunks. The following are the step-by-step instructions: Phase 1: Watermark Marking and Initial Block Selection Figure 2 (a) in the middle, steps 1-4) 1. Pause log processing: First, pause the processing of change logs and locate the current processing node (k3 in the example).

[0034] 2. Generate Low Watermark (L): Request the data source to generate a "low watermark" to mark the starting point of the data range.

[0035] 3. Select initial block: Based on the low watermark, select an initial block result set containing k1, k2, k3, k4, k5, k6 from the data source.

[0036] 4. Generate High Watermark (H): Request the data source to generate a "high watermark", mark the end point of the data range, and complete the delineation of the watermark range.

[0037] Phase Two: Log Recovery and Deduplication Figure 2 (b) in the middle, steps 5-7) 5. Resume log processing: Resume the change log processing flow and continue consuming data from the pause point (k3) onwards.

[0038] 6. Reaching the low watermark and deduplication: When processing reaches the low watermark (L) position, perform a "deduplication" operation on the initial block result set (k1,k2,k3,k4,k5,k6) to remove records that are duplicated in the interval from the low watermark to the high watermark (such as k1,k3), and obtain the deduplicated block result set (k2,k4,k5,k6).

[0039] 7. Reaching the High Watermark: When processing reaches the high watermark (H) position, confirm that the data processing of the watermark interval is complete.

[0040] Phase 3: Output the final data block All records in the change log first flow into the output buffer. Combined with the deduplication results, a unique and complete data block chunk: k2, k4, k5, k6 is finally extracted for subsequent data processing or storage.

[0041] In specific implementation, please refer to Figure 3 , Figure 3 This is a schematic diagram of the overall data synchronization process provided in Embodiment 1 of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method of this application; Figure 3 This demonstrates the distributed data reading architecture of the Source component in Flink CDC (Change Data Capture). The core logic achieves efficient data acquisition under high parallelism through a "Split enumeration-allocation-reading" process. The following are detailed descriptions of each component and process: 1. Core Component Description Data source ( Figure 3 The blue database on the left: the source of the data, which can be understood as an Oracle database.

[0042] SplitEnumerator (including SplitAssigner): This is the "central hub for task splitting and allocation," responsible for dividing the data source into multiple data splits and assigning them to different SourceReaders. The internal SplitAssigner handles the specific split allocation strategy (such as round-robin or load balancing). SourceReader (1 / 2 / 3): These are parallel data readers, each corresponding to a parallel subtask. Internally, they include: SplitFetcher: responsible for fetching the assigned splits; SplitReader: actually reading the split data; RecordEmitter: converting the read data into records that Flink can process and outputting them.

[0043] Data Split: SnapshotSplit: Split based on database snapshots (used for full read phase); BinlogSplit: Split based on database incremental logs (used for incremental read phase).

[0044] 2. Process Interaction (Steps 1-3) Step 1 / 2: Enumeration and Allocation of Full Data Splits (SnapshotSplit) The SplitEnumerator first splits the full data from the data source into SnapshotSplits, and then, via arrow 1 / 2, distributes these splits to SourceReader-1 and SourceReader-2 in a round-robin fashion. The two SourceReaders fetch the splits through SplitFetcher, the SplitReader reads the data, and finally, the data is output through RecordEmitter.

[0045] Step 3: Allocation of Incremental Splits (BinlogSplit) After the full read is complete, SplitEnumerator splits the incremental data into BinlogSplits and allocates them to SourceReader-3 via arrow 3. SourceReader-3 also reads the Binlog data through SplitFetcher and SplitReader, and the incremental changes are output by RecordEmitter.

[0046] In response to a data synchronization request, this embodiment performs data sharding on the full data to be synchronized, obtaining full data blocks. These full data blocks are then round-robin distributed to concurrent threads for full data synchronization. Once the full data synchronization is complete, a lock-free consistency switching strategy is used to shard the incremental data to be synchronized, and incremental data synchronization is performed based on the obtained target data blocks. This lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on a watermarking mechanism. Compared to existing methods that require read locks on the database or table being read during data synchronization, this embodiment enables lock-free data synchronization, preventing business transactions from being blocked.

[0047] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 4 , Figure 4 This is a flowchart illustrating the second embodiment of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method provided in this application. After the step of responding to the data synchronization request, the method further includes the following steps: Step S101: Synchronize all data to be synchronized to the data warehouse tables using the database export tool; It should be noted that the database export tool can be Oracle SQLlldr, and the data warehouse table can be a Hive table.

[0048] Step S102: Write the full amount of data to be synchronized into the financial-grade lake warehouse platform according to the data warehouse table; It should be noted that writing the full data to be synchronized to the financial-grade lake warehouse platform according to the data warehouse table can mean writing the full data to be synchronized from the data warehouse table to the financial-grade lake warehouse platform.

[0049] Step S103: Send the incremental data to be synchronized to the message queue; It should be noted that sending the incremental data to be synchronized to the message queue can be done by using OGG technology during the incremental phase to synchronously send the incremental change data (i.e., the incremental data to be synchronized) in the Oracle (data source) to the Kafka message queue.

[0050] Step S104: Write the incremental data to be synchronized into the financial-grade lake warehouse platform based on the message queue.

[0051] It should be noted that writing the incremental data to be synchronized to the financial-grade lake warehouse platform based on the message queue can be done by writing the incremental data to be synchronized to the financial-grade lake warehouse platform based on the message queue.

[0052] Furthermore, current financial enterprise-level data warehouses primarily process and analyze structured data, with data volumes reaching petabyte (PB) levels in the consumer finance and credit sector. Because they cannot integrate semi-structured and unstructured data, some data cannot be processed and integrated into the warehouse / database. Therefore, to enable the integration of semi-structured and unstructured data, after the step of performing data sharding of the incremental data to be synchronized based on a lock-free consistency switching strategy and synchronizing the incremental data based on the sharding results, the following steps are also included: Obtain the semi-structured and unstructured data to be synchronized; The semi-structured data and the unstructured data are synchronized to the financial-grade lake warehouse platform.

[0053] It should be noted that the semi-structured data includes approval messages, decision messages, etc., and the unstructured data includes voice files, image files, etc. The step of synchronizing the semi-structured data to the financial-grade lakeware platform includes: The semi-structured data is stored in a preset file system, and the file path corresponding to the semi-structured data is recorded. The semi-structured data is parsed based on the file path to obtain structured data; The structured data is synchronized to the financial-grade lake warehouse platform.

[0054] It should be noted that the preset file system can be an SFTP file system. The step of performing structured parsing on the semi-structured data based on the file path to obtain structured data can be achieved by determining the semi-structured data according to the file path and then using the Spark engine to perform structured parsing on the semi-structured data to obtain structured data.

[0055] In practical implementation, semi-structured data such as approval and decision-making messages are stored in real-time in an SFTP file system, and the corresponding file paths are recorded in a relational database (i.e., the data source database). After synchronizing the file paths using a lakeware-integrated financial-grade multi-source heterogeneous data synchronization device, the approval and decision-making messages are simultaneously synchronized into the lake. After entering the lake, the Spark engine is used to parse the data into structured data within minutes before storing it in the warehouse. At this point, the semi-structured data has completed its lakeware storage process within minutes.

[0056] The step of synchronizing the unstructured data to the financial-grade lakeware platform includes: The unstructured data is stored in a preset file system, and the file path corresponding to the unstructured data is recorded. The unstructured data is vectorized based on the file path to obtain vectorized data; The vectorized data is processed according to a preset data processing model to obtain the model processing result; The model processing results are synchronized to the financial-grade lake warehouse platform.

[0057] It should be noted that the preset data processing model can be an AI Python model used to process vectorized data and convert it into structured data.

[0058] In practical implementation, for unstructured data such as voice files and image files, this embodiment pre-stores user-uploaded voice and image files in an SFTP file system and records the corresponding file paths in a relational database (i.e., the data source database). After synchronizing the paths of the voice and image files using a lakeware-integrated financial-grade multi-source heterogeneous data synchronization device, the voice and image files are simultaneously synchronized into the lake. After entering the lake, the Spark engine and parsing algorithms are used to parse the voice and image files into usable vectorized data for data entry into the lake. Then, an AI Python model is used to batch read the corresponding vectorized data for model processing and analysis to obtain model results (structured data), which are then stored in a relational database. Finally, the structured data of the model results is synchronized into the lake and warehouse within minutes.

[0059] In this implementation, the integrated financial-grade lake warehouse platform includes a data warehouse source layer (Hive), a data lake, a data warehouse integration layer, a data warehouse semantic layer, a data warehouse mart layer, and a business report query and display module. It also includes a business master database and a backup synchronization database. The business master database and backup synchronization database are Oracle deployment components. The SFTP file system is the system for storing files in the business system. The integrated financial-grade lake warehouse is based on the Hadoop CDP platform. The backup synchronization database synchronizes data into Kafka using the OGG component configuration. The incremental synchronization of the data warehouse Hive source layer and Kafka into the lake uses the FlinkCDC component and tasks. The data warehouse integration layer, following the data lake, integrates wide tables based on financial domain business themes, such as customer themes, product themes, event themes, and protocol themes. The data warehouse semantic layer processes business semantic indicators based on the integration layer, such as the user's current repayment status and the date of the most recent actual repayment. The data warehouse mart layer abstracts and processes report data with single, unified requirements based on the data from the regular semantic layer. The business report query and display module uses a unified reporting system to configure the corresponding mart layer tables for on-demand display.

[0060] The backup synchronization library uses Oracle SQLlldr to read data files and directly writes them to Hive tables using Flink. Each table in the backup synchronization library is configured with OGG links to send incremental Kafka messages as needed, including messages for adding, deleting, and modifying DML data changes in the configuration tables. FlinkCDC's full incremental process supports switching between full concurrent initialization from Oracle to Iceberg and lock-free full incremental consumption of Kafka. It also supports full read initialization from Hive to Iceberg and lock-free full incremental consumption of customized Kafka messages sent by OGG. The integration layer, semantic layer, and data mart layer in the data warehouse are financial-grade data warehouse layers, with incremental processing logic. Finally, the data in the data mart layer is configured for data reporting applications, providing data value. After semi-structured and unstructured data is fed into the data warehouse, Spark has developed user-defined functions (UDFs) specifically for parsing semi-structured text. Based on the parsed structure of these functions, the resulting data is written to the data warehouse during processing.

[0061] In this embodiment, FlinkCDC utilizes ACID transactions and row-level updates for incremental, minute-level synchronization into the data warehouse, ensuring consistent and accurate data writing. The data warehouse integration and semantic layer processes data using time travel to process data from specific time snapshots of the tables. Both Iceberg and Hive are stored within the Hadoop big data system. The Spark computing engine can directly perform joint queries and processing between streaming Iceberg and offline Hive, achieving unified batch and streaming storage and unified processing by the computing engine. The unified metadata module supports unified management of Hive and Iceberg tables.

[0062] This embodiment also includes a data synchronization and audit verification process from Oracle (data source) to Iceberg. The core logic is to ensure the consistency of the data link through a dual mechanism of "full baseline + incremental sampling". Specifically: 1. Data acquisition and storage link: The source is an Oracle database, and the data is streamed to the Iceberg data lake via a Flink CDC Job (Change Data Capture Job).

[0063] At the same time, Oracle and Iceberg will perform "incremental sampling" on detailed data (such as aa, bb, cc) and total data (statistics of new additions and updates, such as adding 50 and updating 80) based on the create_time and update_time fields at preset intervals (such as 30 minutes), and pass the sampled data to the "audit service".

[0064] 2. Audit services (intermediate, core verification process) The "audit service" is the central hub for verifying data consistency; it receives two types of data simultaneously: Incremental sampling data (details and total snapshots every 30 minutes); T-1 full data from Oracle and Iceberg (the baseline of the full data from the previous day, such as adding 300 or updating 800).

[0065] The verification is divided into two layers: Total data verification: Verify whether the "new" and "update" statistics of the full data are consistent with the total data of the incremental sampling; Detailed total verification: After classifying the data (e.g., category A, category B), verify whether the new and updated statistics match after classification.

[0066] 3. Audit Result Feedback The audit service feeds back the verified total data and detailed total data (such as adding 300 to category A and updating 800 to category B) to the data link, ultimately ensuring the accuracy of detailed data and the consistency of total statistics during the synchronization process from Oracle to Iceberg, and avoiding data loss or errors.

[0067] This embodiment uses a database export tool to synchronize the full data to be synchronized to a data warehouse table; based on the data warehouse table, it writes the full data to be synchronized to a financial-grade lake warehouse platform; it sends the incremental data to be synchronized to a message queue; and based on the message queue, it writes the incremental data to be synchronized to the financial-grade lake warehouse platform. This embodiment uses Oracle SQL Server DROP technology to directly synchronize the full data to a Hive table via offline files. During the incremental phase, OGG technology is used to synchronize the incremental changes in Oracle to Kafka. The synchronization function of FlinkCDC is enhanced and modified to enable full writing from Hive to Iceberg (i.e., the financial-grade lake warehouse platform) and incremental writing from Kafka to Iceberg. The switching between full and incremental data is lock-free, improving overall synchronization efficiency.

[0068] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the lakeware integrated financial-grade multi-source heterogeneous data synchronization method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0069] This application also provides a lakeware-integrated, financial-grade, multi-source heterogeneous data synchronization device. Please refer to [link / reference]. Figure 5 The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization device includes: Response module 10 is used to respond to a data synchronization request and, based on the data synchronization request, perform data fragmentation on the full amount of data to be synchronized to obtain a full data block; The full data synchronization module 20 is used to distribute the full data block to the full data synchronization concurrent thread in a round-robin fashion for full data synchronization; The acquisition module 30 is used to acquire incremental data to be synchronized through a lock-free consistency switching strategy after the full data to be synchronized has been synchronized. The incremental data synchronization module 40 is used to perform data fragmentation on the incremental data to be synchronized based on a lock-free consistency switching strategy after the full data to be synchronized has been synchronized, and to perform incremental data synchronization based on the obtained target data block. The lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on a watermarking mechanism.

[0070] In response to a data synchronization request, this embodiment performs data sharding on the full data to be synchronized, obtaining full data blocks. These full data blocks are then round-robin distributed to concurrent threads for full data synchronization. Once the full data synchronization is complete, a lock-free consistency switching strategy is used to shard the incremental data to be synchronized, and incremental data synchronization is performed based on the obtained target data blocks. This lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on a watermarking mechanism. Compared to existing methods that require read locks on the database or table being read during data synchronization, this embodiment enables lock-free data synchronization, preventing business transactions from being blocked.

[0071] The lakeware-integrated financial-grade multi-source heterogeneous data synchronization device provided in this application, employing the lakeware-integrated financial-grade multi-source heterogeneous data synchronization method described in the above embodiments, can solve the technical problem of existing data synchronization methods that affect business processes due to locking. Compared with the prior art, the beneficial effects of the lakeware-integrated financial-grade multi-source heterogeneous data synchronization device provided in this application are the same as those of the lakeware-integrated financial-grade multi-source heterogeneous data synchronization method provided in the above embodiments, and other technical features in the lakeware-integrated financial-grade multi-source heterogeneous data synchronization device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0072] This application provides a lakeware-integrated financial-grade multi-source heterogeneous data synchronization device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the lakeware-integrated financial-grade multi-source heterogeneous data synchronization method in Embodiment 1 above.

[0073] The following is for reference. Figure 6 This document illustrates a structural diagram of a lakeware-integrated financial-grade multi-source heterogeneous data synchronization device suitable for implementing embodiments of this application. The lakeware-integrated financial-grade multi-source heterogeneous data synchronization device in this application embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 6The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0074] like Figure 6 As shown, the lakeware-based integrated financial-grade multi-source heterogeneous data synchronization device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the lakeware-based integrated financial-grade multi-source heterogeneous data synchronization device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the lakeware-based integrated financial-grade multi-source heterogeneous data synchronization device to exchange data with other devices via wireless or wired communication. Although the figure shows a lakeware-based integrated financial-grade multi-source heterogeneous data synchronization device with various systems, it should be understood that implementing or having all the systems shown is not required. More or fewer systems can be implemented alternatively.

[0075] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0076] The lakeware-integrated financial-grade multi-source heterogeneous data synchronization device provided in this application, employing the lakeware-integrated financial-grade multi-source heterogeneous data synchronization method described in the above embodiments, can solve the technical problem of existing data synchronization methods that affect business processes due to locking. Compared with the prior art, the beneficial effects of the lakeware-integrated financial-grade multi-source heterogeneous data synchronization device provided in this application are the same as those of the lakeware-integrated financial-grade multi-source heterogeneous data synchronization method provided in the above embodiments, and other technical features in this lakeware-integrated financial-grade multi-source heterogeneous data synchronization device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0077] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0078] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0079] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the lakeware integrated financial-grade multi-source heterogeneous data synchronization method in the above embodiments.

[0080] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0081] The aforementioned computer-readable storage medium may be included in a lakeware-based integrated financial-grade multi-source heterogeneous data synchronization device; or it may exist independently and not be assembled into a lakeware-based integrated financial-grade multi-source heterogeneous data synchronization device.

[0082] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include object-oriented programming languages—such as Python, Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0083] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0084] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0085] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described lakeware-integrated financial-grade multi-source heterogeneous data synchronization method. This solves the technical problem of existing data synchronization methods that rely on locking, which impact business processes. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the lakeware-integrated financial-grade multi-source heterogeneous data synchronization method provided in the above embodiments, and will not be elaborated upon here.

[0086] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A method for synchronizing multi-source heterogeneous data at a financial grade based on a lake-warehouse integrated system, characterized in that, The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method includes the following steps: In response to a data synchronization request, the full data to be synchronized is fragmented based on the data synchronization request to obtain a full data block; The full data blocks are distributed in a round-robin fashion to the full data synchronization concurrent thread for full data synchronization; Once the full data to be synchronized is synchronized, the incremental data to be synchronized is sharded based on a lock-free consistency switching strategy, and incremental data synchronization is performed based on the obtained target data blocks. The lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on a watermarking mechanism.

2. The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method as described in claim 1, characterized in that, The steps of performing data fragmentation on the incremental data to be synchronized based on the lock-free consistency switching strategy, and synchronizing incremental data based on the obtained target data blocks, include: Pause the processing of change logs and perform data sharding on the incremental data to be synchronized to obtain initial data blocks; The initial data block is deduplicated to obtain the target data block; Incremental data synchronization is performed based on the target data block.

3. The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method as described in claim 2, characterized in that, The step of deduplicating the initial data block to obtain the target data block includes: Obtain high-watermark markers and restore the change log processing flow; The initial data block is deduplicated based on the paused processing node, the low watermark, and the high watermark marker to obtain the target data block, wherein the paused processing node is the processing node when the change log is paused.

4. The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method as described in claim 1, characterized in that, Following the step of responding to the data synchronization request, the method further includes: The database export tool is used to synchronize all the data to be synchronized to the data warehouse tables. Write the full amount of data to be synchronized into the financial-grade lake warehouse platform according to the data warehouse table; Send the incremental data to be synchronized to the message queue; The incremental data to be synchronized is written to the financial-grade lake warehouse platform based on the message queue.

5. The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method as described in any one of claims 1-4, characterized in that, The target of the data synchronization is a financial-grade lake warehouse platform; After the steps of performing data sharding on the incremental data to be synchronized based on the lock-free consistency switching strategy and synchronizing the incremental data based on the sharding results, when the full data to be synchronized is completed, the method further includes: Obtain the semi-structured and unstructured data to be synchronized; The semi-structured data and the unstructured data are synchronized to the financial-grade lake warehouse platform.

6. The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method as described in claim 5, characterized in that, The step of synchronizing the semi-structured data to the financial-grade lake warehouse platform includes: The semi-structured data is stored in a preset file system, and the file path corresponding to the semi-structured data is recorded. The semi-structured data is parsed based on the file path to obtain structured data; The structured data is synchronized to the financial-grade lake warehouse platform.

7. The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method as described in claim 5, characterized in that, The step of synchronizing the unstructured data to the financial-grade lakeware platform includes: The unstructured data is stored in a preset file system, and the file path corresponding to the unstructured data is recorded. The unstructured data is vectorized based on the file path to obtain vectorized data; The vectorized data is processed according to a preset data processing model to obtain the model processing result; The model processing results are synchronized to the financial-grade lake warehouse platform.

8. A financial-grade multi-source heterogeneous data synchronization device based on a lake-warehouse integrated system, characterized in that, The lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization device includes: The response module is used to respond to a data synchronization request and, based on the data synchronization request, perform data fragmentation on the full amount of data to be synchronized to obtain a full data block; The full data synchronization module is used to distribute the full data blocks in a round-robin fashion to the full data synchronization concurrent thread for full data synchronization; The acquisition module is used to acquire incremental data to be synchronized through a lock-free consistency switching strategy after the full data to be synchronized has been synchronized. The incremental data synchronization module is used to perform data fragmentation on the incremental data to be synchronized based on a lock-free consistency switching strategy after the full data to be synchronized has been synchronized. The incremental data synchronization is performed based on the obtained target data block. The lock-free consistency switching strategy is a snapshot and incremental data collaborative processing strategy based on a watermarking mechanism.

9. A financial-grade multi-source heterogeneous data synchronization device based on a lake-warehouse integrated system, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the lake-warehouse integrated financial-grade multi-source heterogeneous data synchronization method as described in any one of claims 1 to 7.

Images