Data synchronization method, data synchronization system, computer device and medium

Abstract

This application relates to the technical field of big data processing, and in particular to a data synchronization method, a data synchronization system, computer equipment, and media. The method is applied to a data synchronization system, which includes a real-time data warehouse and an offline data warehouse deployed within the same database. The method includes: updating the real-time data warehouse with incremental data obtained from the data source; every first cycle, synchronizing the updated incremental data in the real-time data warehouse as historical data to the offline data warehouse; every second cycle, obtaining full data from the data source, verifying the consistency between the full data and the data already stored in the offline data warehouse, correcting the data already stored in the offline data warehouse based on the verification result, and synchronizing the corrected data in the offline data warehouse to the real-time data warehouse. Using the method of this application can improve the consistency of data between the real-time data warehouse and the offline data warehouse.

Description

Technical Field

[0001] This application relates to the technical field of big data processing, and in particular to a data synchronization method, a data synchronization system, a computer device, and a medium. Background Technology

[0002] In the insurance industry, facing massive amounts of data, offline and real-time data warehouses are often used in conjunction for data storage and analysis. Typically, offline and real-time data warehouses are deployed in different databases. The real-time data warehouse updates incremental data generated by the business system in real time, while the offline data warehouse periodically updates all historical data from the business system. Offline data warehouses possess stronger data processing and analysis capabilities, and are therefore often used (e.g., based on Hadoop) for batch data processing and historical data analysis. Real-time data warehouses, on the other hand, have weaker batch data processing and analysis capabilities, and are therefore primarily used for real-time data processing and analysis.

[0003] Because offline data warehouses are typically updated less frequently, and the existing data in business systems may also change, the freshness of data in offline data warehouses lags behind that of real-time data warehouses. Furthermore, when data in business systems is updated to the real-time data warehouse in real time, data packet loss may occur, which may result in relatively incomplete data in the real-time data warehouse. Therefore, the consistency between data in offline and real-time data warehouses is poor.

[0004] Improving the consistency between real-time and offline data warehouses is an urgent problem that needs to be solved. Summary of the Invention

[0005] Therefore, it is necessary to provide a data synchronization method, data synchronization system, computer equipment, and media that can improve the consistency of data in real-time data warehouses and offline data warehouses, in order to address the above-mentioned technical problems.

[0006] In a first aspect, this application provides a data synchronization method, which is applied to a data synchronization system, the data synchronization system including a real-time data warehouse and an offline data warehouse deployed in the same database, the method comprising:

[0007] The incremental data obtained from the data source will be updated to the real-time data warehouse, where the incremental data is real-time data generated by the business system.

[0008] Every first cycle, the incremental data updated in the real-time data warehouse is synchronized to the offline data warehouse as historical data;

[0009] Every second cycle, a verification task is generated and executed. The verification task is used to obtain full data from the data source and verify the consistency between the full data and the data already stored in the offline data warehouse to obtain the verification result. The data already stored in the offline data warehouse is then corrected based on the verification result. The first cycle is shorter than the second cycle.

[0010] The correction results of the data in the offline data warehouse are synchronized to the real-time data warehouse.

[0011] In one embodiment, the step of obtaining incremental data from the data source and updating the real-time data warehouse with the incremental data includes:

[0012] Obtain initial real-time data generated by the business system from the data source;

[0013] The initial real-time data is preprocessed to obtain preprocessed incremental data;

[0014] The incremental data is updated in the ODS of the real-time data warehouse, and a snapshot of the incremental data is generated and stored; the preprocessing includes at least one of data cleaning, data deduplication, format conversion and standardization.

[0015] In one embodiment, the method further includes:

[0016] Upon receiving a probe request, a target snapshot is determined within the real-time data warehouse, and the abnormal data is compared with the target snapshot to determine if they are consistent. The target snapshot is the snapshot of the incremental data corresponding to the abnormal data, and the abnormal data is the data that the probe request points to, which exists in the ODS of the offline data warehouse and is inconsistent with the historical data corresponding to it in the DWS of the offline data warehouse. The data in the ODS of the offline data warehouse is updated to the DWS every third period, and the third period is longer than the first period.

[0017] If they match, then modify the abnormal data based on the target snapshot;

[0018] If there is a discrepancy, the data in the ODS of the offline data warehouse is modified based on the target snapshot.

[0019] In one embodiment, the method further includes:

[0020] Obtain the total update amount, which is the total amount of incremental data updated to the real-time data warehouse since the last execution of the verification task;

[0021] When the total update amount exceeds the target value, a verification task is generated and executed.

[0022] The target value is determined based on the increment of the target duration and the curve relationship, which represents the relationship between the data increment and the increment of the task duration used to execute the task in each verification task.

[0023] In one embodiment, synchronizing the incremental data updated in the real-time data warehouse to the offline data warehouse includes:

[0024] Based on the list of incremental data recorded in the target update log, update the incremental data to the offline data warehouse;

[0025] The target update log records a list of incremental data updated in the real-time data warehouse after the previous update time, and the update time is the time when the real-time data warehouse updates the incremental data to the offline data warehouse.

[0026] In one embodiment, the process of determining the target update log includes:

[0027] Verify the first update log and the second update log, and determine a new update log as the target update log based on the corresponding non-duplicate incremental data in the first update log and the second update log;

[0028] The first update log records a list of incremental data updates from the data source to the real-time data warehouse through a first channel, and the second update log records a list of incremental data updates from the data source to the real-time data warehouse through a second channel.

[0029] In one embodiment, the method further includes:

[0030] Upon receiving a cross-temporal query request, the offline data warehouse and the real-time data warehouse are queried respectively based on the cross-temporal query request;

[0031] The cross-temporal query request is a query request that targets both the historical data and the real-time data simultaneously.

[0032] The query results for the real-time data warehouse and the offline data warehouse are integrated into a target query result, and the target query result is fed back to the initiator of the cross-temporal query request.

[0033] Secondly, this application also provides a data synchronization system, which includes a data synchronization module, and a real-time data warehouse and an offline data warehouse deployed in the same database, wherein:

[0034] The real-time data warehouse obtains incremental data from real-time data sources for data updates; the incremental data is real-time data generated by the business system.

[0035] The data synchronization module is used to synchronize the incremental data updated in the real-time data warehouse as historical data to the offline data warehouse every first cycle.

[0036] The data synchronization module is further configured to, every second period, obtain full data from the data source, verify the consistency between the full data and the data already stored in the offline data warehouse, obtain the verification result, and correct the data already stored in the offline data warehouse based on the verification result; the second period is longer than the first period;

[0037] The data synchronization module is also used to synchronize the correction results of the data in the offline data warehouse to the real-time data warehouse.

[0038] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program for a data synchronization system, the data synchronization system including a real-time data warehouse and an offline data warehouse deployed in the same database, and the processor executes the computer program to perform the following steps:

[0039] The incremental data obtained from the data source will be updated to the real-time data warehouse, where the incremental data is real-time data generated by the business system.

[0040] Every first cycle, the incremental data updated in the real-time data warehouse is synchronized to the offline data warehouse as historical data;

[0041] Every second cycle, a verification task is generated and executed. The verification task is used to obtain full data from the data source and verify the consistency between the full data and the data already stored in the offline data warehouse to obtain the verification result. The data already stored in the offline data warehouse is then corrected based on the verification result. The first cycle is shorter than the second cycle.

[0042] The correction results of the data in the offline data warehouse are synchronized to the real-time data warehouse.

[0043] Fourthly, this application also provides a computer-readable storage medium storing a computer program for a data synchronization system, the data synchronization system including a real-time data warehouse and an offline data warehouse deployed in the same database, wherein the computer program, when executed by a processor, performs the following steps:

[0044] The incremental data obtained from the data source will be updated to the real-time data warehouse, where the incremental data is real-time data generated by the business system.

[0045] Every first cycle, the incremental data updated in the real-time data warehouse is synchronized to the offline data warehouse as historical data;

[0046] Every second cycle, a verification task is generated and executed. The verification task is used to obtain full data from the data source and verify the consistency between the full data and the data already stored in the offline data warehouse to obtain the verification result. The data already stored in the offline data warehouse is then corrected based on the verification result. The first cycle is shorter than the second cycle.

[0047] The correction results of the data in the offline data warehouse are synchronized to the real-time data warehouse.

[0048] The aforementioned data synchronization method, system, computer equipment, and media, because the first cycle is shorter than the second cycle, allow the real-time data warehouse to frequently update incremental data to the offline data warehouse, maintaining relative consistency between the two. Furthermore, every second cycle, the offline data warehouse compares its stored data with the full data retrieved from the data source, corrects the stored data based on the comparison results, and then synchronizes the corrected data to the real-time data warehouse. This ensures that the data stored in the real-time data warehouse is updated by the corrected data from the offline data warehouse, maintaining consistency between the historical data stored in both warehouses. On one hand, the real-time data warehouse frequently updates real-time incremental data to the offline data warehouse; on the other hand, the offline data warehouse periodically updates its stored data and synchronizes it with the real-time data warehouse, thereby improving the consistency between real-time data and stored historical data between the real-time and offline data warehouses. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a diagram illustrating the application environment of a data synchronization method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a data synchronization method in one embodiment;

[0052] Figure 3This is a schematic diagram of the data warehouse structure in one embodiment;

[0053] Figure 4 This is a schematic diagram of the process of updating incremental data in a real-time data warehouse in one embodiment;

[0054] Figure 5 This is a schematic diagram of the data query process in one embodiment;

[0055] Figure 6 This is a block diagram of a data synchronization system in one embodiment;

[0056] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0058] In the insurance industry, facing massive amounts of data, offline and real-time data warehouses are often used in conjunction for data storage and analysis. Typically, offline and real-time data warehouses are deployed in different databases. The real-time data warehouse updates incremental data generated by the business system in real time, while the offline data warehouse periodically updates all historical data from the business system. Offline data warehouses possess stronger data processing and analysis capabilities, and are therefore often used (e.g., based on Hadoop) for batch data processing and historical data analysis. Real-time data warehouses, on the other hand, have weaker batch data processing and analysis capabilities, and are therefore primarily used for real-time data processing and analysis.

[0059] Because offline data warehouses are typically updated less frequently, and the existing data in business systems may also change, the freshness of data in offline data warehouses lags behind that of real-time data warehouses. Furthermore, when data in business systems is updated to the real-time data warehouse in real time, data packet loss may occur, which may result in relatively incomplete data in the real-time data warehouse. Therefore, the consistency between data in offline and real-time data warehouses is poor.

[0060] The data synchronization method provided in this application embodiment can be applied to, for example... Figure 1 The data synchronization system shown. The data source of the data synchronization system includes at least one business system, such as... Figure 1The diagram shows business systems 1, 2, ..., N; each business system has a corresponding database to store real-time and historical data generated by the system. The data synchronization system includes at least a real-time data warehouse, an offline data warehouse, a data synchronization module, and a query service module. Both the real-time and offline data warehouses can retrieve existing data (already stored data) from business sources, but at different frequencies. The data synchronization module updates the data in both the real-time and offline data warehouses, thereby improving data consistency. The query service module responds to query requests from the initiator, queries the real-time and / or offline data warehouses based on the request, and returns the query results to the initiator.

[0061] In one exemplary embodiment, such as Figure 2 As shown, a data synchronization method is provided, which can be applied to... Figure 1 The following steps are used as an example of a data synchronization system, including steps 210-240:

[0062] Step 210: Update the real-time data warehouse with incremental data obtained from the data source. The incremental data is the real-time data generated by the business system.

[0063] In the embodiments of this application, each business system of the data source is configured to transmit real-time data to the real-time data warehouse through a transmission channel built between the business system and the real-time data warehouse when real-time data is generated. The real-time data warehouse receives the real-time data, performs preprocessing to obtain incremental data, and updates the real-time data warehouse with the incremental data. The incremental data is the part of the real-time data that has changed relative to the historical data already stored in the real-time data warehouse. The incremental data also includes data that has not been stored in the real-time data warehouse.

[0064] Step 220: Every first cycle, synchronize the incremental data updated in the real-time data warehouse as historical data to the offline data warehouse.

[0065] In this embodiment of the application, the first period can be preset by the user; in this embodiment, the first period can be set to a duration of three hours; that is, every three hours is an update time; starting from the previous update time, every three hours is the next update time. At each update time, the incremental data updated in the real-time data warehouse between the previous update time and the current update time is updated to the offline data warehouse, so that the offline data warehouse updates these incremental data simultaneously.

[0066] Step 230: Every second cycle, generate and execute a verification task. The verification task is used to obtain full data from the data source and verify the consistency between the full data and the data already stored in the offline data warehouse to obtain the verification result. Based on the verification result, correct the data already stored in the offline data warehouse. The first cycle is shorter than the second cycle.

[0067] In this embodiment, the second period can be preset by the user or dynamically adjusted according to real-time changes in relevant conditions; this embodiment does not specifically limit this. However, the second period must be longer than the first period. In one possible implementation, the second period can be set to one month. That is, starting from the last update time, a verification task is created and executed once a month. Since the real-time data generated within the business system may be modifications to historical data already stored in the business system's database, or new data may be entered and stored, the historical data already stored within the business system may also be constantly changing.

[0068] During the verification process, the existing data (all stored data) obtained from the data source is compared with the historical data stored in the offline database to obtain the verification results. The verification results include data that is already stored in the offline data warehouse and needs to be modified in the offline data warehouse, as well as data that needs to be newly written into the offline data warehouse. Based on the verification results, the offline data warehouse is corrected, that is, the offline data warehouse is updated using the full amount of data from the data source, so that the data in the offline data warehouse is consistent with the data from the data source.

[0069] Step 240: Synchronize the correction results of the data in the offline data warehouse to the real-time data warehouse.

[0070] In this embodiment of the application, the correction result is the stored data obtained after the offline data warehouse has been completely corrected based on the verification result. When the offline data warehouse completes the data correction, the correction result is synchronized to the real-time data warehouse, so that the data stored in the real-time data warehouse is synchronously corrected based on the correction result of the offline data warehouse. Since the real-time data warehouse is synchronously corrected based on the corrected data in the offline data warehouse, the data stored in the real-time data warehouse and the offline data warehouse can remain consistent after both the real-time data warehouse and the offline data warehouse have completed the correction.

[0071] Furthermore, when the offline data warehouse completes the data correction based on the verification results, it indicates that the data pointed to by the verification results is not abnormal. At this time, the real-time data warehouse synchronously corrects the data based on the correction results of the offline data warehouse, which can improve the stability of the data when it is corrected and reduce the probability of anomalies when the data already stored in the real-time data warehouse is corrected.

[0072] In the aforementioned data synchronization method, since the first period is shorter than the second period, the real-time data warehouse can frequently synchronize updated incremental data to the offline data warehouse using the first period as the interval. This ensures relative consistency between the offline and real-time data warehouses in terms of real-time data. Furthermore, every second period, the offline data warehouse compares its stored data with the full data obtained from the data source and corrects the stored data based on the comparison results. The corrected data is then synchronized to the real-time data warehouse, allowing the data stored in the real-time data warehouse to be updated by the correction results from the offline data warehouse. This ensures consistency between the historical data stored in the offline and real-time data warehouses. On one hand, the real-time data warehouse frequently updates real-time incremental data to the offline data warehouse; on the other hand, the offline data warehouse periodically updates its stored data and synchronizes it with the real-time data warehouse, thereby improving the consistency between real-time data and stored historical data between the real-time and offline data warehouses.

[0073] In one embodiment, the real-time data warehouse is built on Paimon, while the offline data warehouse is built on Hadoop. Paimon is an open-source lake warehouse unified storage format that provides efficient stream and batch data processing capabilities, primarily focusing on data storage, management, and efficient interaction with the computing engine, supporting real-time and offline data analysis and processing. Hadoop is an open-source big data processing framework and a comprehensive ecosystem, primarily comprising the Hadoop Distributed File System (HDFS) for storing large-scale data and the MapReduce computing model for distributed computing.

[0074] The real-time data warehouse and offline data warehouse in the data synchronization system have the same structure, such as... Figure 3 As shown, both real-time and offline data warehouses include an ODS (Operational Data Store), a DIM (Dimension Table), and a DWD (Data Warehouse Detail).

[0075] ODS (Optical Data Store) is the data storage layer in a data warehouse system that is closest to the source system. It primarily stores relatively raw, less processed data. Data in the ODS layer is characterized by frequent updates, reflecting the latest state of the business system in a timely manner. ODS is mainly used to provide operational reports and concise analyses to help management make timely business decisions.

[0076] Dimension tables (DIMs) are tables in a data warehouse used to describe the various characteristics of analytical objects (such as time, location, and product). DIMs typically contain external attributes related to the business theme, enabling multi-dimensional queries and data analysis. The data in dimension tables is relatively stable and updated infrequently.

[0077] The DWD layer is the detailed data layer in the data warehouse that has undergone initial cleaning and standardization, but still retains a high degree of granularity and detail. This layer of data is primarily used to support subsequent data aggregation operations and complex analyses.

[0078] The DWS layer refers to the data layer extracted from the DWD layer and further processed and summarized. The data granularity of the DWS layer is lower than that of the DWD layer, making it more suitable for report display and further business analysis. The data in this layer can be divided into multiple smaller data models or subject areas according to different business needs to meet the query requirements of different user groups. Typically, the data in the DWS layer is updated every other day; that is, the update frequency of the DWS layer is greater than the first cycle but less than the second cycle.

[0079] Furthermore, in one embodiment, such as Figure 4 As shown, step 210 may specifically include steps 211-213, wherein:

[0080] Step 211: Obtain the initial real-time data generated by the business system from the data source;

[0081] Step 212: Preprocess the initial real-time data to obtain preprocessed incremental data; preprocessing includes at least one of the following: data cleaning, data deduplication, format conversion, and standardization.

[0082] Specifically, the data source links to the initial real-time data generated by various business systems. This initial real-time data is the most raw data, which may contain duplicate data, and the format of each initial real-time data may not be uniform. Therefore, the initial real-time data must be preprocessed before it is updated and stored in the real-time data warehouse, so that the data updated and stored in the real-time data warehouse by the data source is standardized incremental data.

[0083] Since real-time data warehouses collect data from various business systems of data sources, and the data in real-time data warehouses is updated in a streaming manner, network fluctuations may cause some initial real-time data to experience packet loss or other faults, resulting in incomplete or lost data. Therefore, in this embodiment of the application, a dual-channel synchronous transmission method for initial real-time data is adopted to further verify the consistency and validity of the initial real-time data.

[0084] In one possible implementation, two data channels exist between the data source and the real-time data warehouse: a first channel and a second channel. Initial real-time data is simultaneously transmitted to the real-time data warehouse through both channels, and the real-time data warehouse stores the initial real-time data received from the first and second channels respectively. The real-time data warehouse maintains an update log, which records a list of data items written, along with the operation steps and timestamps for each item in the list.

[0085] Furthermore, an update log is created for each of the data transmitted through the first and second channels. The first update log records a list of incremental data updates from the data source to the real-time data warehouse via the first channel, and the second update log records a list of incremental data updates from the data source to the real-time data warehouse via the second channel. The real-time data warehouse verifies the consistency of the initial real-time data received from the two channels. After confirming the consistency of the initial real-time data, it preprocesses the consistent initial valid data to obtain standardized incremental data. The methods for determining the validity of the initial real-time data include, but are not limited to: comparing the hash values ​​of two sets of initial real-time data to determine if they are consistent; and comparing the file sizes of two sets of initial real-time data, determining consistency if the file sizes are the same.

[0086] By using dual-channel updates to obtain initial real-time data, data loss caused by network problems can be avoided, and log backtracking can also improve the synchronization and reliability of data updates.

[0087] Step 213: Update the incremental data to the ODS of the real-time data warehouse and generate a snapshot of the incremental data for storage.

[0088] Specifically, the ODS is used to store the standardized incremental data obtained through the preprocessing in step 212. After corresponding analysis and processing, the data in the ODS is updated to the DWD, DIM, and DWS. Furthermore, when incremental data is written to the ODS of the real-time data warehouse, a snapshot for backup is generated for each piece of incremental data. The snapshot is identical to the corresponding incremental data but stored in a different location; for example, the incremental data is stored in a high-performance storage address, such as memory or SSD, while the snapshot is stored in a lower-performance storage address, such as HDFS or object storage. Of course, the incremental data can also be stored in HDFS or object storage, and the snapshot can also be stored in memory or SSD; this embodiment does not impose specific limitations. Snapshots are used for data tracing and verification in case of data inconsistencies, providing factual evidence for data tracing.

[0089] Furthermore, in one embodiment, the data in the offline data warehouse ODS is updated to the DWS every third cycle; wherein the third cycle is greater than or equal to the first cycle and less than the second cycle; the third cycle can be set by the user, for example, it can be 24 hours.

[0090] In step 220, the real-time data warehouse updates the incremental data in the ODS to the ODS of the offline data warehouse every first period. That is, the offline data updated from the real-time data warehouse to the offline data warehouse does not need to be preprocessed again. In one example, the real-time data warehouse last updated incremental data to the offline data warehouse at 10:00. With a first period of three hours, the real-time data warehouse should update the incremental data in the ODS of the offline data warehouse again at 13:00, and the incremental data updated to the offline data warehouse is the incremental data updated by the real-time data warehouse from 10:00 to 13:00.

[0091] Furthermore, the first period can be a fixed value set by the user, such as three hours; the first period can also be dynamically adjusted, for example, starting from the moment when the real-time data warehouse last updated incremental data to the offline data warehouse, counting the number of updated records, which is the number of incremental data records updated by the real-time data warehouse; when the number of updated records is greater than or equal to the number of incremental data records last updated by the real-time data warehouse to the offline data warehouse, the current moment is determined as the update moment, that is, step 220 is executed, that is, the real-time data warehouse immediately updates incremental data to the offline data warehouse.

[0092] Because the frequency of real-time data generation varies among different business systems in the data source, there may be bursts of real-time data generation during certain periods. If the real-time data warehouse maintains a fixed interval for updating incremental data to the offline data warehouse during these periods, it may reduce the timeliness of the data in the offline data warehouse, which means that the amount of inconsistent data between the real-time and offline data warehouses will increase. Therefore, by counting the number of incremental data updates in the real-time data warehouse since the last update, the timing of the next update to the offline data warehouse can be dynamically adjusted, so that the data in the offline data warehouse can also maintain a certain level of freshness during periods of busy incremental data updates.

[0093] Furthermore, in step 220, each time the real-time data warehouse updates incremental data to the offline data warehouse, it updates the incremental data to the offline data warehouse according to the list of incremental data recorded in the target update log; wherein, the target update log records the list of new incremental data in the real-time data warehouse after the previous update time, and the update time is the time when the real-time data warehouse updates the incremental data to the offline data warehouse.

[0094] Specifically, the real-time data warehouse records the initial real-time data list received from the first channel in the first update log and the initial real-time data list received from the second channel in the second update log. The real-time data warehouse verifies the consistency of the initial real-time data received from the first and second channels, preprocesses the consistent initial real-time data to obtain incremental data, and deletes duplicate initial real-time data.

[0095] In one possible implementation, the real-time data warehouse checks the contents of the first and second update logs since the last update moment, and determines a new update log as the target update log based on the corresponding non-duplicate incremental data in the first and second update logs.

[0096] In another possible implementation, the real-time data warehouse forms a new update log based on each incremental data record obtained through preprocessing; the target update log is obtained by extracting data from the previous update time in the new update log.

[0097] The real-time data warehouse updates incremental data to the offline data warehouse based on the target update log. On the one hand, the data in the update log has been checked and verified, which can avoid updating duplicate and / or incomplete incremental data to the offline data warehouse. On the other hand, the list of incremental data updated from the data source to the real-time data warehouse is recorded in the target update log. Updating incremental data to the offline data warehouse based on the target update log can avoid missing any incremental data, thereby improving the consistency of data in the real-time data warehouse and the offline data warehouse.

[0098] Furthermore, in step 230, the verification task is generated once every second cycle; the second cycle can be a fixed length set by the user, such as one month; that is, after the last verification task is generated, the next verification task will be generated after a one-month interval. In one possible implementation, the generation time of the verification task can also be dynamically adjusted based on the total amount of incremental data updated in the real-time data warehouse; since the verification task retrieves full data from the data source and compares it with the data already stored in the offline data warehouse, and since the full data in the data source is constantly accumulating newly generated real-time data after the last verification task is executed, the amount of data that needs to be verified each time the verification task is executed is actually greater than the amount of data in the previous verification task (not considering the deletion of historical data), which results in a large amount of data to be verified by the verification task, and usually requires a long time, such as 24 hours.

[0099] During the data verification process, the computing and analytical capabilities of the offline data warehouse are significantly reduced, and its response time to query requests decreases considerably. Therefore, it is essential to control the duration of each verification task. In one embodiment, a verification task is generated and executed when the total update amount exceeds a target value; the target value is determined based on the increment of the target duration and a curve relationship, where the curve relationship characterizes the relationship between the data increment in each verification task and the increment of the task duration used for execution.

[0100] Specifically, at the moment each execution task is generated, the total amount of incremental data updated in real-time is counted. When the total update amount reaches a set target value, a verification task is generated. The target value is determined based on the impact of the total update amount on the task duration. The task duration of each verification task is calculated, along with the increment of incremental data for each verification task relative to the previous one. Linear analysis and fitting are then performed to obtain a curve showing the relationship between the increment of incremental data and the increase in task duration. The user sets the target increase in duration, and based on the fitted relationship curve, the increment of incremental data corresponding to the target duration increase is determined as the target amount.

[0101] In one embodiment, since the ODS in the offline data warehouse stores incremental data updated by the real-time data warehouse, and the DWS retrieves data from the ODS for updates every third period (which is longer than the first period), if a user's query request simultaneously requests data from both the ODS and the DWS, the query results returned to the user may be inconsistent. In this case, the user may initiate a probe request to verify the inconsistent data. Therefore, the data synchronization method provided in this application embodiment further includes steps 310-330, as follows... Figure 5 As shown, where:

[0102] Step 310: Upon receiving a probe request, determine the target snapshot in the real-time data warehouse and compare the abnormal data with the target snapshot to see if they are consistent.

[0103] Specifically, the target snapshot is a snapshot of the incremental data corresponding to the anomalous data. The anomalous data is data that exists in the ODS of the offline data warehouse and is inconsistent with the historical data in the DWS of the offline data warehouse, as indicated by the probe request. Further, based on the attributes of the anomalous data, the latest snapshot corresponding to the anomalous data is searched in the real-time data warehouse as the target snapshot. The attributes of the data are parameters that can characterize the data's identity, such as at least one of name, producer, source, and unique code. In this embodiment, the type of parameters in the data attributes is not specifically limited.

[0104] Step 320: If they match, modify the data in the DWS of the offline data warehouse based on the target snapshot;

[0105] Step 330: If there is a discrepancy, modify the abnormal data based on the target snapshot.

[0106] If the abnormal data stored in the ODS in the offline data warehouse is consistent with the data corresponding to the target snapshot stored in the real-time data warehouse, it means that the abnormal data is correct. The inconsistency in query results is caused by the incremental data stored in the ODS not being updated to the DWS in a timely manner. In this case, the historical data in the DES in the offline data warehouse can be modified based on the target snapshot.

[0107] If the anomalous data stored in the ODS of the offline data warehouse is inconsistent with the data corresponding to the target snapshot stored in the real-time data warehouse, there are two possible scenarios: First, an error occurred during the process of updating incremental data to the offline data warehouse by the real-time data system, resulting in the anomalous data. Second, the data in the ODS updated from the real-time data warehouse to the offline data warehouse did not contain errors, but the business source subsequently generated new data corresponding to that data. The real-time data warehouse updated the new data, but because the first cycle interval had not yet been reached, the real-time data warehouse did not synchronize the new data to the offline data warehouse. Therefore, in cases where anomalous data is inconsistent with the target snapshot, the anomalous data is modified based on the target snapshot to ensure consistency and high freshness between the data in the offline and real-time data warehouses.

[0108] Furthermore, in related technologies, since real-time data warehouses and offline data warehouses are configured in two independent databases, when a requester performs a data query, it typically organizes separate query commands for each of the two databases based on the query request. The real-time data warehouse usually responds to the query commands for real-time data, while the offline data warehouse responds to the query commands for historical data. However, when faced with query requests involving both real-time and historical data across different time and space periods, in order for the business system to obtain query results for both real-time and historical data simultaneously, it is usually necessary to merge the data in the real-time data warehouse into the offline data warehouse. The offline data warehouse then responds to the cross-time and space query request and returns the query results. Because of this data merging process, the query efficiency for cross-time and space data is relatively low.

[0109] In the data synchronization system of this application embodiment, when a query request for real-time data is received, the real-time data warehouse is queried based on the query request; when a query request for historical data is received, the offline data warehouse is queried based on the query request. Further, when a cross-temporal query request is received, both the offline data warehouse and the real-time data warehouse are queried based on the cross-temporal query request; wherein, the cross-temporal query request is a query request for both historical data and real-time data simultaneously; the query results for the real-time data warehouse and the offline data warehouse are integrated into a target query result, and the target query result is fed back to the initiator of the cross-temporal query request, thereby enabling the initiator to simultaneously obtain query results for both real-time and historical data.

[0110] In the data synchronization system of this application embodiment, since the real-time data warehouse and the offline data warehouse are configured in the same database, both the offline and real-time data warehouses can be queried simultaneously without needing to organize query commands for each data warehouse separately. Because the process of this application embodiment reduces the database merging process, it can improve the efficiency of cross-temporal and spatial queries. In the embodiments of this application, the initiator can query and retrieve data from the database configuring the real-time and offline data warehouses using API (Application Programming Interface) or SQL (Structured Query Language).

[0111] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0112] Based on the same inventive concept, this application also provides a data synchronization system for implementing the data synchronization method described above. The solution provided by this system is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data synchronization system embodiments provided below can be found in the limitations of the data synchronization method described above, and will not be repeated here.

[0113] In one exemplary embodiment, such as Figure 6 As shown, a data synchronization system is provided, including a data synchronization module, and a real-time data warehouse and an offline data warehouse deployed in the same database, wherein:

[0114] Real-time data warehouses acquire incremental data from real-time data sources for data updates; the incremental data is real-time data generated by business systems.

[0115] The data synchronization module is used to synchronize incremental data updated in the real-time data warehouse as historical data to the offline data warehouse every first cycle.

[0116] The data synchronization module is also used to retrieve full data from the data source every second cycle, verify the consistency between the full data and the data already stored in the offline data warehouse, obtain the verification result, and correct the data already stored in the offline data warehouse based on the verification result; the second cycle is longer than the first cycle;

[0117] The data synchronization module is also used to synchronize the correction results of data in the offline data warehouse to the real-time data warehouse.

[0118] In one embodiment, the data synchronization module is specifically used for:

[0119] Obtain initial real-time data generated by the business system from the data source;

[0120] Preprocess the initial real-time data to obtain preprocessed incremental data;

[0121] Incremental data is updated to the ODS of the real-time data warehouse, and a snapshot of the incremental data is generated and stored; preprocessing includes at least one of the following: data cleaning, data deduplication, format conversion, and standardization.

[0122] In one embodiment, the data synchronization module is further configured to:

[0123] Upon receiving a probe request, a target snapshot is determined within the real-time data warehouse, and the abnormal data is compared with the target snapshot for consistency. The target snapshot is a snapshot of the incremental data corresponding to the abnormal data. The abnormal data is the data that the probe request points to, which exists in the ODS of the offline data warehouse and is inconsistent with the corresponding historical data in the DWS of the offline data warehouse. Within this context, the data in the ODS of the offline data warehouse is updated to the DWS every third cycle, and the third cycle is longer than the first cycle.

[0124] If they match, modify the abnormal data based on the target snapshot;

[0125] If there is a discrepancy, the data in the offline data warehouse's ODS will be modified based on the target snapshot.

[0126] In one embodiment, the data synchronization module is further configured to:

[0127] Get the total update amount, which is the total amount of incremental data updated to the real-time data warehouse since the last execution of the generation and verification task;

[0128] When the total update amount exceeds the target value, a verification task is generated and executed.

[0129] The target value is determined based on the target update time, as well as the update time and the execution time of each historical verification task.

[0130] In one embodiment, the data synchronization module is further configured to:

[0131] Based on the list of incremental data recorded in the target update log, update the incremental data to the offline data warehouse;

[0132] The target update log records a list of incremental data updated in the real-time data warehouse after the previous update time. The update time is the moment when the real-time data warehouse updates the incremental data to the offline data warehouse.

[0133] In one embodiment, the data synchronization module is further configured to:

[0134] Verify the first update log and the second update log, and determine the new update log as the target update log based on the corresponding non-duplicate incremental data in the first update log and the second update log;

[0135] The first update log records the list of incremental data that the data source updates to the real-time data warehouse through the first channel, and the second update log records the list of incremental data that the data source updates to the real-time data warehouse through the second channel.

[0136] In one embodiment, the data synchronization system further includes a query service module, wherein the query service module is specifically used for:

[0137] Upon receiving a cross-temporal query request, query both the offline data warehouse and the real-time data warehouse based on the cross-temporal query request.

[0138] Among them, cross-temporal query requests are query requests that simultaneously target historical data and real-time data;

[0139] The query results from the real-time data warehouse and the offline data warehouse are integrated into the target query result, and the target query result is fed back to the initiator of the cross-temporal query request.

[0140] Each module in the aforementioned data synchronization system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0141] Furthermore, the data synchronization system in this embodiment is based on Paimon as a real-time data warehouse and combines it with Hadoop's offline data storage and processing capabilities to achieve data synchronization and integrated querying between the two. Through unified data management and real-time batch data mixing processing, high availability and consistency of the data warehouse are achieved. The architecture and configuration of each module in the data synchronization system are described in detail below.

[0142] Real-time data processing module:

[0143] The real-time data processing module uses Paimon as its core technology, enabling real-time access, preprocessing, and analysis of data from multiple sources to meet real-time data response requirements. Real-time data from data sources flows into Paimon via Kafka and undergoes real-time data preprocessing (preliminary data cleaning and formatting). Paimon's table storage mode supports low-latency data writing and high-concurrency querying.

[0144] Stream processing and computation: Real-time data aggregation, computation, and analysis are achieved using SQL-like commands. Leveraging Paimon's data stream processing capabilities, the real-time data warehouse enables second-level data analysis support in insurance operations, laying the foundation for subsequent data integration.

[0145] Offline data processing module:

[0146] The offline data processing module, based on Hadoop's distributed architecture, is responsible for the historical storage and analysis of batch data. Offline data is extracted in batches from business systems, and data consistency is ensured through data cleaning and normalization. Data within the offline data warehouse is stored in HDFS, supporting large-scale data storage and efficient batch processing. Tools such as Hive enable batch analysis of historical data stored in the offline data warehouse, suitable for periodic data mining needs. Hadoop's batch processing capabilities allow the offline data warehouse to support the management and in-depth analysis of long-term data, providing a data source for real-time data integration.

[0147] Data synchronization module: The data synchronization module is responsible for integrating and consistently updating real-time and offline data to ensure data consistency in Paimon and Hadoop.

[0148] Incremental data capture: Paimon’s real-time data updates are captured through the CDC (Change Data Capture) mechanism and updated to the offline data warehouse every three hours.

[0149] Batch data backflow: Periodically extract existing data from the business system, compare all data in the business system with the data in the offline data warehouse, correct the data, and update and synchronize it to Paimon to achieve two-way data flow, so that real-time queries can use the latest data.

[0150] The data synchronization module supports the integration of offline and real-time data through scheduled batch updates and incremental synchronization, effectively improving the depth and accuracy of real-time data analysis.

[0151] Query service layer:

[0152] The query service layer serves as a unified data query entry point for business applications, supporting seamless joint queries of real-time and historical data.

[0153] Cross-warehouse query interface design: Support business needs for accessing real-time and historical data through API and SQL interfaces.

[0154] Query optimization: Dynamically optimize query paths based on data type (real-time or historical), and improve query performance by using partitioned indexes and caching strategies.

[0155] Real-time-offline integrated query: The query engine supports joint queries between Paimon and Hadoop, enabling businesses to obtain both real-time and historical data simultaneously.

[0156] The query service layer integrates the data query capabilities of Paimon and Hadoop, enabling insurance businesses to access both real-time and offline data simultaneously on a single interface, thus achieving efficient integrated querying of real-time and historical data.

[0157] In summary, the data synchronization system of this application integrates real-time analysis and historical trend data through Paimon's real-time processing and Hadoop's offline storage. Paimon can store and compute real-time data streams in real time, while Hadoop provides support for larger-scale data storage and batch processing. The two coordinate data consistency through a synchronization module. On the one hand, the data synchronization system of this application achieves seamless integration of real-time and historical data, improving the real-time performance of data analysis. On the other hand, through the joint optimization of Paimon and Hadoop, data consistency in real-time and offline warehouses is ensured, and the development and maintenance costs of real-time and offline data warehouses are reduced, improving the scalability and ease of operation and maintenance of the data warehouse system. Furthermore, the query service layer ultimately enables the business layer to access all data in a unified manner, thereby meeting the comprehensive data analysis needs of insurance business.

[0158] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a data synchronization method applied to a data synchronization system, which includes a real-time data warehouse and an offline data warehouse deployed within the same database. The display unit of the computer device is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0159] Those skilled in the art will understand that Figure 7The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0160] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program applied to a data synchronization system, the processor executing the computer program to implement the steps as described in the above-described data synchronization method embodiment.

[0161] In one embodiment, a computer-readable storage medium is provided, on which a computer program for a data synchronization system is stored. The data synchronization system includes a real-time data warehouse and an offline data warehouse deployed in the same database. When the computer program is executed by a processor, it implements the steps as described in the above-described data synchronization method embodiment.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0163] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0164] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0165] The above embodiments merely illustrate several implementation methods of this application, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of this application's patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data synchronization method, characterized in that, The method is applied to a data synchronization system, which includes a real-time data warehouse and an offline data warehouse deployed in the same database. The method includes: The incremental data obtained from the data source will be updated to the real-time data warehouse, where the incremental data is real-time data generated by the business system. Every first cycle, the incremental data updated in the real-time data warehouse is synchronized to the offline data warehouse as historical data; Every second cycle, a verification task is generated and executed. The verification task is used to obtain full data from the data source and verify the consistency between the full data and the data already stored in the offline data warehouse to obtain the verification result. The data already stored in the offline data warehouse is then corrected based on the verification result. The first cycle is shorter than the second cycle. Synchronize the correction results of the data in the offline data warehouse to the real-time data warehouse; Upon receiving a probe request, a target snapshot is determined within the real-time data warehouse, and the abnormal data is compared with the target snapshot to determine if they are consistent. The target snapshot is the snapshot of the incremental data corresponding to the abnormal data, and the abnormal data is the data that the probe request points to, which exists in the ODS of the offline data warehouse and is inconsistent with the historical data corresponding to it in the DWS of the offline data warehouse. The data in the ODS of the offline data warehouse is updated to the DWS every third period, and the third period is longer than the first period. If they match, then modify the abnormal data based on the target snapshot; If there is a discrepancy, the data in the ODS of the offline data warehouse is modified based on the target snapshot.

2. The method according to claim 1, characterized in that, The step of obtaining incremental data from the data source and updating the real-time data warehouse with the incremental data includes: Obtain initial real-time data generated by the business system from the data source; The initial real-time data is preprocessed to obtain preprocessed incremental data; The incremental data is updated in the ODS of the real-time data warehouse, and a snapshot of the incremental data is generated and stored; the preprocessing includes at least one of data cleaning, data deduplication, format conversion and standardization.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the total update amount, which is the total amount of incremental data updated to the real-time data warehouse since the last execution of the verification task; When the total update amount exceeds the target value, a verification task is generated and executed. The target value is determined based on the increment of the target duration and the curve relationship, which represents the relationship between the data increment and the increment of the task duration used to execute the task in each verification task.

4. The method according to claim 1, characterized in that, The step of synchronizing the incremental data updated in the real-time data warehouse to the offline data warehouse includes: Based on the list of incremental data recorded in the target update log, update the incremental data to the offline data warehouse; The target update log records a list of incremental data updated in the real-time data warehouse after the previous update time, and the update time is the time when the real-time data warehouse updates the incremental data to the offline data warehouse.

5. The method according to claim 4, characterized in that, The process of determining the target update log includes: Verify the first update log and the second update log, and determine a new update log as the target update log based on the corresponding non-duplicate incremental data in the first update log and the second update log; The first update log records a list of incremental data updates from the data source to the real-time data warehouse through a first channel, and the second update log records a list of incremental data updates from the data source to the real-time data warehouse through a second channel.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: Upon receiving a cross-temporal query request, the offline data warehouse and the real-time data warehouse are queried respectively based on the cross-temporal query request; The cross-temporal query request is a query request that targets both the historical data and the real-time data simultaneously. The query results for the real-time data warehouse and the offline data warehouse are integrated into a target query result, and the target query result is fed back to the initiator of the cross-temporal query request.

7. A data synchronization system, characterized in that, The data synchronization system includes a data synchronization module, as well as a real-time data warehouse and an offline data warehouse deployed in the same database, wherein: The real-time data warehouse obtains incremental data from real-time data sources for data updates; the incremental data is real-time data generated by the business system. The data synchronization module is used to synchronize incremental data generated in the real-time data warehouse as historical data to the offline data warehouse every first cycle. The data synchronization module is further configured to, every second period, obtain full data from the data source, verify the consistency between the full data and the data already stored in the offline data warehouse, obtain the verification result, and correct the data already stored in the offline data warehouse based on the verification result; the second period is longer than the first period; The data synchronization module is also used to synchronize the correction results of the data in the offline data warehouse to the real-time data warehouse; The data synchronization module is further configured to, upon receiving a probe request, determine a target snapshot within the real-time data warehouse and compare the abnormal data with the target snapshot for consistency; the target snapshot is a snapshot of the incremental data corresponding to the abnormal data, and the abnormal data is data that exists in the ODS of the offline data warehouse and is inconsistent with the historical data corresponding to it in the DWS of the offline data warehouse, as indicated by the probe request; wherein, the data in the ODS of the offline data warehouse is updated to the DWS every third period, the third period being longer than the first period; if consistent, the abnormal data is modified based on the target snapshot; if inconsistent, the data in the ODS of the offline data warehouse is modified based on the target snapshot.

8. The system according to claim 7, characterized in that, The data synchronization module is also used for: Obtain initial real-time data generated by the business system from the data source; The initial real-time data is preprocessed to obtain preprocessed incremental data; The incremental data is updated in the ODS of the real-time data warehouse, and a snapshot of the incremental data is generated and stored. The preprocessing includes at least one of the following: data cleaning, data deduplication, format conversion, and standardization.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images