Data synchronization method, data synchronization system, computer equipment and medium

Abstract

The invention relates to the technical field of big data processing, in particular to a data synchronization method, a data synchronization system, computer equipment and a medium. The method is applied to a data synchronization system, the data synchronization system comprises a real-time data warehouse and an offline data warehouse which are deployed in the same database, and the method comprises the following steps: updating incremental data acquired from a data source to the real-time data warehouse, and updating the incremental data to the offline data warehouse every first period; taking the updated incremental data in the real-time data warehouse as historical data and synchronizing the historical data to the offline data warehouse; and every other second period, obtaining full data from the data source, checking the consistency of the full data and the stored data in the offline data warehouse, correcting the stored data in the offline data warehouse according to a checking result, and synchronizing a correction result of the data in the offline data warehouse to the real-time data warehouse. By adopting the method disclosed by the invention, the consistency of the data in the real-time data warehouse and the offline data warehouse can be improved.

Description

Technical Field

[0001] This application relates to the technical field of big data processing, and particularly to a data synchronization method, a data synchronization system, a computer device, and a medium. Background Art

[0002] In the insurance industry, in the face of a large amount of data, an offline data warehouse and a real-time data warehouse are often used in cooperation for data storage and analysis; usually, the offline data warehouse and the real-time data warehouse are respectively deployed in different databases. The real-time data warehouse updates the incremental data generated by the business system in real time, and the offline data warehouse regularly updates the full amount of historical data from the business system. Among them, the offline data warehouse has strong data processing and analysis capabilities, so an offline data warehouse (such as based on Hadoop) is often used for batch data processing and historical data analysis; while the real-time data warehouse has weak batch data processing and data analysis capabilities, so a real-time data warehouse is often used for real-time data processing and analysis.

[0003] Since the update frequency of the offline data warehouse is usually low, and the stock data in the business system may also change, the freshness of the data in the offline data warehouse lags behind that of the real-time data warehouse. And when the data in the business system is updated to the real-time data warehouse in real time, data packets may be lost, which may cause the data in the real-time data warehouse to be relatively incomplete; therefore, the consistency of the data in the offline data warehouse and the real-time data warehouse is poor.

[0004] How to improve the consistency of the data in the real-time data warehouse and the offline data warehouse is an urgent problem to be solved. Summary of the Invention

[0005] Based on this, in view of the above technical problems, it is necessary to provide a data synchronization method, a data synchronization system, a computer device, and a medium that can improve the consistency of the data in the real-time data warehouse and the offline data warehouse.

[0006] In a first aspect, this application provides a data synchronization method. The method is applied to a data synchronization system, and the data synchronization system includes a real-time data warehouse and an offline data warehouse deployed in the same database. The method includes:

[0007] Updating the incremental data obtained from the data source to the real-time data warehouse, where the incremental data is the real-time data generated by the business system;

[0008] Every first period, synchronize the incremental data updated in the real-time data warehouse to the offline data warehouse as historical data;

[0009] Every second period, a verification task is generated and executed. The verification task is used to obtain all data from a data source and verify the consistency between the all data and the data stored in the offline data warehouse, obtain a verification result, and correct the data stored in the offline data warehouse according to the verification result; the first period is less than the second period;

[0010] Synchronize the correction result of the data in the offline data warehouse to the real-time data warehouse.

[0011] In one embodiment, the obtaining incremental data from a data source and updating the incremental data to the real-time data warehouse includes:

[0012] Obtain the initial real-time data generated by a business system from the data source;

[0013] Preprocess the initial real-time data to obtain preprocessed incremental data;

[0014] Update the incremental data to the ODS of the real-time data warehouse and generate a snapshot of the incremental data for storage; the preprocessing includes at least one of data cleaning, data deduplication, format conversion, and standardization processing.

[0015] In one embodiment, the method further includes:

[0016] When a probing request is received, determine a target snapshot in the real-time data warehouse and compare whether the abnormal data is consistent with the target snapshot; the target snapshot is the snapshot of the incremental data corresponding to the abnormal data, and the abnormal data is the data that exists in the ODS of the offline data warehouse pointed to by the probing request and is inconsistent with the corresponding historical data in the DWS of the offline data warehouse; wherein, the data in the ODS of the offline data warehouse is updated to the DWS every third period, and the third period is greater than the first period;

[0017] If they are consistent, modify the abnormal data based on the target snapshot;

[0018] If they are inconsistent, modify the data in the ODS of the offline data warehouse based on the target snapshot.

[0019] In one embodiment, the method further includes:

[0020] Obtain the total update amount, which is the total amount of the incremental data updated to the real-time data warehouse since the last execution of generating the verification task;

[0021] When the total update amount is greater than a target value, generate and execute a verification task;

[0022] Wherein, the target value is determined based on the increment of the target duration and the curve relationship, and the curve relationship characterizes the relationship between the data increment and the increment of the task duration used for each execution of the verification task.

[0023] In one embodiment, the synchronizing the increment data updated in the real-time data warehouse to the offline data warehouse includes:

[0024] Updating the increment data to the offline data warehouse according to the list of the increment data recorded in the target update log;

[0025] Wherein, the target update log records the list of each increment data updated in the real-time data warehouse after the previous update moment, and the update moment is the moment when the real-time data warehouse updates the increment data to the offline data warehouse.

[0026] In one embodiment, the process of determining the target update log includes:

[0027] Verifying the first update log and the second update log, and determining a new update log as the target update log based on the non-repetitive corresponding increment data in the first update log and the second update log;

[0028] Wherein, the first update log records the list of the increment data updated by the data source to the real-time data warehouse through the first channel, and the second update log records the list of the increment data updated by the data source to the real-time data warehouse through the second channel.

[0029] In one embodiment, the method further includes:

[0030] When receiving a cross-time and space query request, querying the offline data warehouse and the real-time data warehouse respectively based on the cross-time and space query request;

[0031] Wherein, the cross-time and space query request is a query request for both the historical data and the real-time data;

[0032] Integrating the query results for the real-time data warehouse and the offline data warehouse respectively into a target query result, and feeding back the target query result to the initiator of the cross-time and space query request.

[0033] In a second aspect, the present application further provides a data synchronization system, the data synchronization system includes a data synchronization module, and a real-time data warehouse and an offline data warehouse deployed in the same database, wherein:

[0034] The real-time data warehouse obtains incremental data from real-time data sources for data update; the incremental data is real-time data generated by a business system;

[0035] The data synchronization module is configured to, every first period, synchronize the updated incremental data in the real-time data warehouse to the offline data warehouse as historical data;

[0036] The data synchronization module is further configured to, every second period, obtain full-volume data from the data source, check the consistency between the full-volume data and the data stored in the offline data warehouse to obtain a check result, and correct the data stored in the offline data warehouse according to the check result; the second period is greater than the first period;

[0037] The data synchronization module is further configured to synchronize the correction result of the data in the offline data warehouse to the real-time data warehouse.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, where the memory stores a computer program applied to a data synchronization system, and the data synchronization system includes a real-time data warehouse and an offline data warehouse deployed in the same database. When the processor executes the computer program, the following steps are implemented:

[0039] Update the incremental data obtained from the data source to the real-time data warehouse, where the incremental data is real-time data generated by a business system;

[0040] Every first period, synchronize the updated incremental data in the real-time data warehouse to the offline data warehouse as historical data;

[0041] Every second period, generate and execute a check task, where the check task is used to obtain full-volume data from the data source, check the consistency between the full-volume data and the data stored in the offline data warehouse to obtain a check result, and correct the data stored in the offline data warehouse according to the check result; the first period is less than the second period;

[0042] Synchronize the correction result of the data in the offline data warehouse to the real-time data warehouse.

[0043] In a fourth aspect, the present application further provides a computer-readable storage medium, on which a computer program applied to a data synchronization system is stored. The data synchronization system includes a real-time data warehouse and an offline data warehouse deployed in the same database. When the computer program is executed by a processor, the following steps are implemented:

[0044] Update the incremental data obtained from the data source to the real-time data warehouse, where the incremental data is real-time data generated by a business system;

[0045] Every first period, synchronize the updated incremental data in the real-time data warehouse to the offline data warehouse as historical data;

[0046] Every second period, generate and execute a verification task. The verification task is used to obtain the full amount of data from the data source, verify the consistency between the full amount of data and the data stored in the offline data warehouse, obtain a verification result, and correct the data stored in the offline data warehouse according to the verification result; the first period is less than the second period;

[0047] Synchronize the correction result of the data in the offline data warehouse to the real-time data warehouse.

[0048] In the above data synchronization method, data synchronization system, computer device and medium, since the first period is less than the second period, the real-time data warehouse can synchronize and update the updated incremental data to the offline data warehouse at high frequency at intervals of the first period. In terms of real-time data, this makes the offline data warehouse and the real-time data warehouse maintain relative consistency; further, every second period, the offline data warehouse verifies the stored data with the full amount of data obtained from the data source, corrects the stored data according to the verification result, and further synchronizes the correction result obtained after the correction to the real-time data warehouse, so that the data stored in the real-time data warehouse can be updated by the correction result of the offline data warehouse, so that the historical data stored in the offline data warehouse and the real-time data warehouse can be kept consistent; on the one hand, the real-time data warehouse synchronizes and updates the real-time incremental data to the offline data warehouse at high frequency, and on the other hand, the offline data warehouse regularly updates the stored data and synchronizes and updates it with the real-time data warehouse, so as to improve the consistency of the real-time data and the stored historical data between the real-time data warehouse and the offline data warehouse. Description of the Drawings

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.

[0050] Figure 1 It is an application environment diagram of the data synchronization method in an embodiment;

[0051] Figure 2 It is a flow diagram of the data synchronization method in an embodiment;

[0052] Figure 3Schematic diagram of the structure of a data warehouse in an embodiment;

[0053] Figure 4 Schematic diagram of the process for updating incremental data in a real-time data warehouse in an embodiment;

[0054] Figure 5 Schematic diagram of the process for data query in an embodiment;

[0055] Figure 6 Block diagram of the structure of a data synchronization system in an embodiment;

[0056] Figure 7 Internal structure diagram of a computer device in an embodiment. Detailed implementation manners

[0057] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application.

[0058] In the insurance industry, in the face of a large amount of data, an offline data warehouse and a real-time data warehouse are often used in cooperation for data storage and analysis; usually, the offline data warehouse and the real-time data warehouse are respectively deployed in different databases, and the real-time data warehouse updates the incremental data generated by the business system in real time, and the offline data warehouse periodically updates the full amount of historical data from the business system. Among them, the offline data warehouse has strong data processing and analysis capabilities, so an offline data warehouse (such as based on Hadoop) is mostly used for batch data processing and historical data analysis; while the real-time data warehouse has weak batch data processing capabilities and data analysis capabilities, so a real-time data warehouse is mostly used for real-time data processing and analysis.

[0059] Since the update frequency of the offline data warehouse is usually low, and the stock data in the business system may also change, the freshness of the data in the offline data warehouse lags behind that of the real-time data warehouse. Moreover, when the data in the business system is updated to the real-time data warehouse in real time, data packet loss may occur, which may lead to relatively incomplete data in the real-time data warehouse; therefore, the consistency of the data in the offline data warehouse and the real-time data warehouse is poor.

[0060] The data synchronization method provided by the embodiments of the present application can be applied to a data synchronization system as Figure 1 shown. Among them, the data source of the data synchronization system includes at least one business system, such as Figure 1The business systems shown, such as business system 1, business system 2... business system N; among them, a corresponding database is configured in each business system to store the real-time data and historical data generated by the business system. The data synchronization system at least includes a real-time data warehouse, an offline data warehouse, a data synchronization module, and a query service module; among them, both the real-time data warehouse and the offline data warehouse can obtain the stock data (already stored data) from the business source, but the frequencies of obtaining the stock data are different. The data synchronization module is used to update the data in the real-time data warehouse and the offline data warehouse, so as to improve the consistency of the data in the real-time data warehouse and the offline data warehouse. The query service module responds to the query request of the initiator, queries the real-time data warehouse and / or the offline data warehouse according to the query request, and feeds back the query result to the initiator.

[0061] In an exemplary embodiment, as Figure 2 shown, a data synchronization method is provided. Taking the data synchronization system in Figure 1 as an example, the method includes the following steps 210-step 240, where:

[0062] Step 210: Update the incremental data obtained from the data source to the real-time data warehouse, and the incremental data is the real-time data generated by the business system.

[0063] For the embodiments of the present application, each business system of the data source is configured to, when generating real-time data, transmit the real-time data to the real-time data warehouse through the transmission channel established between the business system and the real-time data warehouse. The real-time data warehouse receives the real-time data and performs preprocessing to obtain incremental data, and updates the incremental data in the real-time data warehouse; among them, the incremental data is the part of the real-time data that changes relative to the historical data already stored in the real-time data warehouse, and the incremental data also includes data that has not been stored in the real-time data warehouse.

[0064] Step 220: Every first period, synchronize the updated incremental data in the real-time data warehouse to the offline data warehouse as historical data.

[0065] For the embodiments of the present application, the first period can be preset by the user; in the embodiments of the present application, the first period can be set to a duration of three hours; that is, every three hours is an update moment; starting from the previous update moment, every three hours is the next update moment. At each update moment, the incremental data updated in the real-time data warehouse between the previous update moment and the current update moment is updated to the offline data warehouse, so that the offline data warehouse updates these incremental data at the same time.

[0066] Step 230: Generate and execute a verification task every second period. The verification task is used to obtain all data from the data source and verify the consistency between the all data and the data already stored in the offline data warehouse, obtain a verification result, and correct the data already stored in the offline data warehouse according to the verification result; the first period is less than the second period.

[0067] For the embodiments of the present application, the second period can also be preset by the user or dynamically adjusted according to relevant conditions that change in real time. The embodiments of the present application do not specifically limit this; however, the second period should be greater than the first period. In one possible implementation, the second period can be set to one month. That is, starting from the previous update time, a verification task is created and executed every month. Since the real-time data generated in the business system may be a change to the historical data already stored in the database of the business system or new data is entered and stored; therefore, the historical data already stored in the business system may also be constantly changing.

[0068] During the execution of the verification task, the stock data (all stored data) obtained from the data source is verified against the historical data already stored in the offline database to obtain a verification result; the verification result includes the data that has been stored in the offline data warehouse and the data that needs to be modified in the offline data warehouse, and also includes the data that needs to be newly written into the offline data warehouse; the offline data warehouse is corrected based on the verification result, that is, the all data of the data source is used to update the offline data warehouse, so that the data in the offline data warehouse is consistent with the data in the data source.

[0069] Step 240: Synchronize the correction result of the data in the offline data warehouse to the real-time data warehouse.

[0070] For the embodiments of the present application, the correction result is the data already stored in the offline data warehouse after being corrected based on the verification result; when the offline data warehouse completes data correction, the correction result is synchronized into the real-time data warehouse, so that the data already stored in the real-time data warehouse is synchronously corrected based on the correction result of the offline data warehouse; since the real-time data warehouse is synchronously corrected based on the corrected data in the offline data warehouse, after the real-time data warehouse and the offline data warehouse are both corrected, the data already stored in the real-time data warehouse and the offline data warehouse can be kept consistent.

[0071] Furthermore, when the offline data warehouse completes data correction based on the verification result, it indicates that the data pointed to by the verification result is normal; at this time, the real-time data warehouse synchronously corrects the data based on the correction result of the offline data warehouse, which can improve the stability when the data is corrected and reduce the probability of anomalies occurring when the data already stored in the real-time data warehouse is corrected.

[0072] In the above data synchronization method, since the first cycle is smaller than the second cycle, the real-time data warehouse can synchronize the updated incremental data to the offline data warehouse at a high frequency with the first cycle as an interval, which makes the offline data warehouse and the real-time data warehouse maintain relative consistency in terms of real-time data; further, every second cycle, the offline data warehouse verifies the stored data with the full data obtained from the data source, and corrects the stored data according to the verification result, and further synchronizes the corrected result obtained after the correction to the real-time data warehouse, so that the data stored in the real-time data warehouse can be updated by the correction result of the offline data warehouse, so that the historical data stored in the offline data warehouse and the real-time data warehouse can be consistent; on the one hand, the real-time data warehouse synchronizes the real-time incremental data to the offline data warehouse at a high frequency, and on the other hand, the offline data warehouse regularly updates the stored data and synchronizes it with the real-time data warehouse, so that the consistency of the real-time data and the stored historical data between the real-time data warehouse and the offline data warehouse is improved.

[0073] In one embodiment, the real-time data warehouse is built based on Paimon, and the offline data warehouse is built based on Hadoop; Paimon is an open source lake-warehouse integrated storage format; it can provide efficient stream-batch integrated data processing capabilities, mainly focusing on data storage, management and efficient interaction with computing engines, and supporting real-time and offline analysis and processing of data. Hadoop is an open source big data processing framework and a comprehensive ecosystem, which mainly includes the distributed file system HDFS (Hadoop Distributed File System) for storing large-scale data, and the MapReduce computing model for distributed computing.

[0074] The structures of the real-time data warehouse and offline data warehouse in the data synchronization system are the same, such as Figure 3 As shown in the figure, the structures of the real-time data warehouse and the offline data warehouse both include ODS (Operational Data Store), DIM (Dimension Table) and DWD (Data Warehouse Detail), where:

[0075] ODS is the data storage layer closest to the source system in the data warehouse system. It mainly stores relatively original and less processed data. The data in the ODS layer is characterized by frequent updates and can reflect the latest status of the business system in a timely manner. ODS is mainly used to provide operation reports and brief analysis to help management make timely business decisions.

[0076] A DIM is a table in a data warehouse that describes various characteristics of analysis objects (such as time, location, products, etc.). A DIM usually contains external attributes related to business themes and is used to implement multi-dimensional queries and data analysis. The data in the dimension table is relatively stable and has a low update frequency.

[0077] The DWD layer is the layer of detailed data that has been preliminarily cleaned in the data warehouse. The data has completed preliminary cleaning and standardization processing, but still maintains a high granularity and level of detail. The data at this layer is mainly used to support subsequent data aggregation operations and complex analysis.

[0078] The DWS layer refers to the data layer extracted from the DWD layer and further processed and summarized. The data granularity of the DWS layer is lower than that of the DWD layer and is more suitable for report display and further business analysis. The data at this layer can be sliced into multiple small data models or subject areas according to different business requirements to meet the query needs of different user groups. Usually, the data in the DWS layer is updated once every day, that is, the update frequency of the DWS layer is greater than the first period and less than the second period.

[0079] Further, in one of the embodiments, as Figure 4 shown, step 210 may specifically include steps 211 - 213, where:

[0080] Step 211: Obtain the initial real-time data generated by the business system from the data source;

[0081] Step 212: Preprocess the initial real-time data to obtain the preprocessed incremental data; the preprocessing includes at least one of data cleaning, data deduplication, format conversion, and standardization processing.

[0082] Specifically, the data source links the initial real-time data generated by each business system. The initial real-time data is the most primitive data, which may contain duplicate data, and the formats of the initial real-time data may not be unified. Therefore, the initial real-time data needs to be preprocessed before being updated and stored in the real-time data warehouse, so that the data updated and stored in the real-time data warehouse by the data source is all standardized incremental data.

[0083] Since the real-time data warehouse collects data from each business system of the data source and the data in the real-time data warehouse is updated in real-time streaming, due to network fluctuations, there may be faults such as packet loss in some of the initial real-time data during this process, resulting in incomplete / missing data. Therefore, in the embodiments of the present application, a dual-channel synchronous transmission method for the initial real-time data is adopted to further verify the consistency and effectiveness of the initial real-time data.

[0084] In a possible implementation, there are two data channels between the data source and the real-time data warehouse, namely the first channel and the second channel; the initial real-time data is transmitted to the real-time data warehouse through both the first channel and the second channel simultaneously, and the real-time data warehouse stores the initial real-time data received based on the first channel and the second channel respectively. The real-time data warehouse has an update log, which records the list of each piece of data written, as well as the operation steps and timestamps for each piece of data in the list.

[0085] Furthermore, for the data transmitted through the first channel and the data transmitted through the second channel, there is a corresponding update log respectively. The first update log records the list of incremental data updated by the data source to the real-time data warehouse through the first channel, and the second update log records the list of incremental data updated by the data source to the real-time data warehouse through the second channel. The real-time data warehouse checks the consistency of the initial real-time data received from the two channels. After determining the consistency of the initial real-time data, it preprocesses the consistent initial valid data to obtain standardized incremental data. Among them, the methods for determining the validity of the initial real-time data include but are not limited to: respectively comparing the hash values of the two pieces of initial real-time data to determine whether the two pieces of initial real-time data are consistent through the hash values; comparing the file sizes of the two pieces of initial real-time data and determining that the two pieces of initial real-time data are consistent when the file sizes are the same.

[0086] The method of obtaining the initial real-time data through dual-channel updates can avoid data loss caused by network problems, and at the same time can improve the synchronization and reliability of data updates through log backtracking.

[0087] Step 213: Update the incremental data to the ODS of the real-time data warehouse and generate a snapshot of the incremental data for storage.

[0088] Specifically, the ODS is used to store the standardized incremental data obtained through preprocessing in step 212. After the data in the ODS undergoes corresponding analysis and processing, it is updated to DWD, DIM, and DWS. Moreover, when the incremental data is written into the ODS of the real-time data warehouse, a snapshot for backup is generated for each piece of incremental data. The snapshot is the same as the corresponding incremental data, but the storage location is different; for example, the incremental data is stored in a high-performance storage address, such as memory or SSD, while the snapshot is stored in a storage address with lower performance, such as HDFS or object storage. Of course, the incremental data can also be stored in HDFS or object storage, and the snapshot can also be stored in memory or SSD. This application embodiment does not make specific limitations. The snapshot is used for data traceability and verification in case of data inconsistency, providing a factual basis for data traceability.

[0089] Further, in one embodiment, the data in the ODS of the offline data warehouse is updated to the DWS every third period; wherein, the third period is greater than or equal to the first period and less than the second period; the third period can be set by the user, for example, it can be 24 hours.

[0090] In step 220, the real-time data warehouse updates the incremental data updated in the ODS to the ODS of the offline data warehouse every first period, that is, the offline data updated from the real-time data warehouse to the offline data warehouse does not need to be preprocessed again. In an example, the last time the real-time data warehouse updated incremental data to the offline data warehouse was at 10:00 sharp. When the first period is three hours, the real-time data warehouse should update the incremental data to the ODS of the offline data warehouse again at 13:00 sharp, and the incremental data updated to the offline data warehouse is the incremental data updated by the real-time data warehouse from 10:00 sharp to 13:00.

[0091] Further, the first period can be a fixed value set by the user, such as three hours; the first period can also be dynamically adjusted. For example, starting from the time when the real-time data warehouse last updated incremental data to the offline data warehouse, count the number of updated records. The number of updated records is the number of incremental data updated by the real-time data warehouse; when the number of updated records is greater than or equal to the number of incremental data updated by the real-time data warehouse to the offline data warehouse last time, determine the current time as the update time, that is, execute step 220, which means the real-time data warehouse immediately updates the incremental data to the offline data warehouse.

[0092] Since the frequencies of generating real-time data by each business system of the data source are different, in some time periods, real-time data may burst. If the real-time data warehouse still maintains a fixed interval of the first period to update incremental data to the offline data warehouse at this stage, it may cause the timeliness of the data in the offline data warehouse to decrease, that is, the amount of inconsistent data in the real-time data warehouse and the offline data warehouse will increase; therefore, by counting the number of incremental data updated in the real-time data warehouse since the last update, the next update time to the offline data warehouse is dynamically adjusted, so that the data in the offline data warehouse can also maintain a certain freshness during the busy stage of incremental data update.

[0093] Further, in step 220, when the real-time data warehouse updates incremental data to the offline data warehouse each time, it updates the incremental data to the offline data warehouse according to the list of incremental data recorded in the target update log; wherein, the target update log records the list of new incremental data of the real-time data warehouse after the previous update time, and the update time is the time when the real-time data warehouse updates incremental data to the offline data warehouse.

[0094] Specifically, the real-time data warehouse records the initial real-time data list received from the first channel in the first update log, and records the initial real-time data list received from the second channel in the second update log. The real-time data warehouse verifies the consistency of the initial real-time data received from the first channel and the second channel, preprocesses the determined consistent initial real-time data to obtain incremental data, and deletes the duplicate initial real-time data.

[0095] In a possible implementation manner, the real-time data warehouse verifies the content recorded in the first update log and the second update log after the last update moment, and determines a new update log as the target update log based on the non-duplicate incremental data corresponding in the first update log and the second update log.

[0096] In another possible implementation manner, the real-time data warehouse forms a new update log based on each incremental data record obtained by preprocessing; in the new update log, it is intercepted based on the last update moment to obtain the target update log.

[0097] The real-time data warehouse updates the incremental data into the offline data warehouse according to the target update log. On the one hand, the data in the update log has been verified and can avoid updating duplicate and / or incomplete incremental data into the offline data warehouse; on the other hand, the list of incremental data updated from the data source to the real-time data warehouse is recorded in the target update log. Updating the incremental data into the offline data warehouse according to the target update log can avoid missing any incremental data, thereby improving the data consistency between the real-time data warehouse and the offline data warehouse.

[0098] Furthermore, in step 230, the generation frequency of the verification task is once every second period; the second period can be a fixed length set by the user. For example, the second period is one month; that is to say, when the last verification task was generated, the next verification task generation moment is one month later. In a possible implementation manner, the generation moment of the verification task can also be dynamically adjusted based on the total amount of incremental data updated in the real-time data warehouse; since the verification task obtains the full amount of data from the data source and verifies it with the data already stored in the offline data warehouse, after the last execution of the verification task, the full amount of data in the data source is always continuously accumulating newly generated real-time data. Therefore, in fact, the amount of data to be verified in each execution of the verification task is greater than that of the previous verification task (without considering the deletion of historical data), which results in a relatively large amount of data to be verified in the verification task and usually takes a long time, such as 24 hours.

[0099] During the process of performing the verification task for data verification, the computing and analysis capabilities of the offline data warehouse are greatly reduced, and the response ability to query requests will decrease significantly. Therefore, it is necessary to control the task duration of a single execution of the verification task. In one embodiment, when the update total amount is greater than the target value, a verification task is generated and executed; wherein, the target value is determined based on the increment of the target duration and the curve relationship, and the curve relationship represents the relationship between the data increment and the increment of the task duration used in previous verification tasks.

[0100] Specifically, starting from the generation moment of each execution task generation, the total update amount of the increment data updated in the real-time data is statistically calculated. When the total update amount reaches the set target value, it is determined to generate a verification task. Wherein, the target value is determined based on the influence relationship of the total update amount on the task duration. The task duration of each verification task is statistically calculated, and the increment of the increment data of each verification task relative to the previous verification task is also statistically calculated. Then, linear analysis and fitting are performed to obtain the relationship curve between the increment of the increment data and the increase amount of the task duration. The user sets the increase amount of the target duration, and then, according to the determined fitting relationship curve, determines the increment of the increment data corresponding to the increase amount of the target duration as the target amount.

[0101] In one embodiment, since the ODS in the offline data warehouse stores the increment data updated from the real-time data warehouse, and the DWS is updated from the ODS every third period, and since the third period is greater than the first period; therefore, if the query request initiated by the user requests to query the data in both the ODS and DWS in the offline data warehouse at the same time, the query results fed back to the user may be inconsistent. At this time, the user may initiate an exploration request for the inconsistent data to request verification of the data. Therefore, the data synchronization method provided in the embodiments of the present application further includes steps 310-step 330, as Figure 5 shown, wherein:

[0102] Step 310, when receiving the exploration request, determine the target snapshot in the real-time data warehouse, and compare whether the abnormal data is consistent with the target snapshot;

[0103] Specifically, the target snapshot is the snapshot of the increment data corresponding to the abnormal data, and the abnormal data is the data that exists in the ODS of the offline data warehouse pointed to by the exploration request and is inconsistent with the corresponding historical data in the DWS of the offline data warehouse. Further, according to the attributes of the abnormal data, the snapshot with the latest time corresponding to the abnormal data is found in the real-time data warehouse. Among them, the attributes of the data are parameters that can represent the identity of the data, such as at least one of name, producer, source, and unique code. In the embodiments of the present application, the types of parameters in the attributes of the data are not specifically limited.

[0104] Step 320: If they are consistent, modify the data in the DWS of the offline data warehouse based on the target snapshot;

[0105] Step 330: If they are inconsistent, modify the abnormal data based on the target snapshot.

[0106] If the abnormal data stored in the ODS of the offline data warehouse is consistent with the data corresponding to the target snapshot stored in the real-time data warehouse, it indicates that the abnormal data is correct and the inconsistent query result is caused by the incremental data stored in the ODS not being updated to the DWS in time; at this time, only modify the historical data in the DES of the offline data warehouse based on the target snapshot.

[0107] If the abnormal data stored in the ODS of the offline data warehouse is inconsistent with the data corresponding to the target snapshot stored in the real-time data warehouse, there may be two situations; the first situation is that during the process of the real-time data system updating the incremental data to the offline data warehouse, an error occurred in the data, resulting in the appearance of abnormal data; the second situation is that there is no error in the data when it is updated from the real-time data warehouse to the ODS of the offline data warehouse, but the business source generates new data for this data at a later time; the real-time data warehouse updates the new data, but since the interval of the first cycle has not been reached, the real-time data warehouse does not synchronously update the new data to the offline data warehouse. Therefore, when the abnormal data is inconsistent with the target snapshot, modify the abnormal data based on the target snapshot, so that the data in the offline data warehouse is consistent with that in the real-time data warehouse and has a high freshness.

[0108] Furthermore, in the related technology, since the real-time data warehouse and the offline data warehouse are configured in two independent databases; when the requester conducts a data query, usually, query commands for the two databases are respectively organized based on the query request, and then usually, the real-time data warehouse responds to the query command for real-time data, and the offline data warehouse responds to the query command for historical data; and when facing a query request for cross-time and space data that involves both real-time data and historical data, in order to enable the business system to obtain the query results of both real-time data and historical data simultaneously, usually, the data in the real-time data warehouse needs to be merged into the offline data warehouse, and then the offline data warehouse responds to the cross-time and space query request and feeds back the query result. In this process, due to the existence of the data merging process, the query efficiency for cross-time and space data is relatively low.

[0109] In the data synchronization system according to the embodiments of the present application, when a query request for real-time data is received, the real-time data warehouse is queried based on the query request; when a query request for historical data is received, the offline data warehouse is queried based on the query request. Further, when a cross-time-and-space query request is received, the offline data warehouse and the real-time data warehouse are respectively queried based on the cross-time-and-space query request; wherein, the cross-time-and-space query request is a query request for both historical data and real-time data; the query results for the real-time data warehouse and the offline data warehouse are integrated into a target query result, and the target query result is fed back to the initiator of the cross-time-and-space query request, so that the initiator can obtain the query results of real-time data and historical data simultaneously.

[0110] In the data synchronization system according to the embodiments of the present application, since the real-time data warehouse and the offline data warehouse are configured in the same database, it is possible to query the offline data warehouse and the real-time data warehouse simultaneously without separately organizing query commands for the two data warehouses; since the process of the embodiments of the present application reduces the process of database merging, the efficiency of cross-time-and-space query can be improved. In the embodiments of the present application, the initiator can query and call data from the database configuring the real-time data warehouse and the offline data warehouse through an API (Application Programming Interface) or SQL (Structured Query Language).

[0111] It should be understood that although the steps in the flowcharts involved in the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed alternately or in turn with at least a part of other steps or steps or stages in other steps.

[0112] Based on the same inventive concept, the embodiments of the present application also provide a data synchronization system for implementing the data synchronization method involved above. The implementation solutions for solving problems provided by this system are similar to the implementation solutions described in the above method. Therefore, the specific limitations in one or more embodiments of the data synchronization system provided below can refer to the limitations on the data synchronization method in the above text and will not be repeated here.

[0113] In an exemplary embodiment, asFigure 6 As shown in the figure, a data synchronization system is provided, including a data synchronization module, a real-time data warehouse and an offline data warehouse deployed in the same database, where:

[0114] The real-time data warehouse obtains incremental data from the real-time data source for data update; the incremental data is the real-time data generated by the business system;

[0115] The data synchronization module is used to synchronize the updated incremental data in the real-time data warehouse to the offline data warehouse as historical data every first period;

[0116] The data synchronization module is also used to obtain the full amount of data from the data source every second period, check the consistency between the full amount of data and the data stored in the offline data warehouse, obtain the check result, and correct the data stored in the offline data warehouse according to the check result; the second period is greater than the first period;

[0117] The data synchronization module is also used to synchronize the correction result of the data in the offline data warehouse to the real-time data warehouse.

[0118] In one embodiment, the data synchronization module is specifically used for:

[0119] Obtain the initial real-time data generated by the business system from the data source;

[0120] Preprocess the initial real-time data to obtain the preprocessed incremental data;

[0121] Update the incremental data to the ODS of the real-time data warehouse, and generate a snapshot of the incremental data for storage; the preprocessing includes at least one of data cleaning, data deduplication, format conversion, and standardization processing.

[0122] In one embodiment, the data synchronization module is also used for:

[0123] When receiving a probe request, determine the target snapshot in the real-time data warehouse, and compare whether the abnormal data is consistent with the target snapshot; the target snapshot is the snapshot of the incremental data corresponding to the abnormal data, and the abnormal data is the data that exists in the ODS of the offline data warehouse pointed to by the probe request and is inconsistent with the corresponding historical data in the DWS of the offline data warehouse; therein, the data in the ODS of the offline data warehouse is updated to the DWS every third period, and the third period is greater than the first period;

[0124] If they are consistent, modify the abnormal data based on the target snapshot;

[0125] If they are inconsistent, modify the data in the ODS of the offline data warehouse based on the target snapshot.

[0126] In one embodiment, the data synchronization module is further configured to:

[0127] Obtain the total amount of updates, where the total amount of updates is the total amount of incremental data updated to the real-time data warehouse since the last execution of generating the verification task;

[0128] When the total amount of updates is greater than the target value, generate and execute a verification task;

[0129] Wherein, the target value is determined based on the target update time, and the update time and the execution times of each historical verification task.

[0130] In one embodiment, the data synchronization module is further configured to:

[0131] Update the incremental data to the offline data warehouse according to the list of incremental data recorded in the target update log;

[0132] Wherein, the target update log records the list of each incremental data updated in the real-time data warehouse after the last update moment, and the update moment is the moment when the real-time data warehouse updates the incremental data to the offline data warehouse.

[0133] In one embodiment, the data synchronization module is further configured to:

[0134] Verify the first update log and the second update log, and determine a new update log as the target update log based on the non-repeated incremental data corresponding in the first update log and the second update log;

[0135] Wherein, the first update log records the list of incremental data updated by the data source to the real-time data warehouse through the first channel, and the second update log records the list of incremental data updated by the data source to the real-time data warehouse through the second channel.

[0136] In one embodiment, the data synchronization system further includes a query service module, where the query service module is specifically configured to:

[0137] When receiving a cross-time and space query request, query the offline data warehouse and the real-time data warehouse respectively based on the cross-time and space query request;

[0138] Wherein, the cross-time and space query request is a query request for historical data and real-time data at the same time;

[0139] Integrate the query results for the real-time data warehouse and the offline data warehouse respectively into a target query result, and feedback the target query result to the initiator of the cross-time and space query request.

[0140] Each module in the above data synchronization system can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above modules can be embedded in the processor of a computer device in hardware form or be independent of it, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each of the above modules.

[0141] Furthermore, the data synchronization system in the embodiments of this application is based on Paimon as a real-time data warehouse, and combines the offline data storage and processing capabilities of Hadoop to achieve data synchronization and integrated query between the two. Through unified data management and real-time batch data hybrid processing, the high availability and consistency of the data warehouse are achieved; the architectures and configuration methods of each module in the data synchronization system are elaborated in detail below.

[0142] Real-time data processing module:

[0143] The real-time data processing module uses Paimon as the core technology and can achieve real-time access, preprocessing, and analysis of data in multi-source data to meet the real-time data response requirements. The real-time data of the data source flows into Paimon through Kafka and undergoes real-time data preprocessing (preliminary data cleaning and formatting). The table storage mode of Paimon supports low-latency writing and high-concurrency query of data.

[0144] Stream processing and computing: Aggregation computing and analysis of real-time data are implemented based on SQL-like commands. Through the data stream processing ability of Paimon, the real-time data warehouse can achieve second-level data analysis support in the insurance business, laying a foundation for subsequent data integration.

[0145] Offline data processing module:

[0146] The offline data processing module is based on the distributed architecture of Hadoop and is responsible for historical storage and analysis of batch data. Offline data is batch-extracted from the business system, and data consistency is ensured through data cleaning and normalization. The data in the offline data warehouse is stored in HDFS, which supports large-scale data storage and efficient batch processing. Through tools such as Hive, batch analysis of historical data stored in the offline data warehouse can be achieved, which is suitable for periodic data mining requirements. The batch processing ability of Hadoop enables the offline data warehouse to support long-term data management and in-depth analysis, providing a data source for real-time data integration.

[0147] Data synchronization module: The data synchronization module is responsible for the integration and consistent update of real-time data and offline data, ensuring data consistency in Paimon and Hadoop.

[0148] Incremental data capture: Capture the real-time data updates of Paimon through the CDC (Change Data Capture) mechanism, and update them to the offline data warehouse every three hours.

[0149] Batch data backflow: Regularly extract the stock data from the business system, compare all the data in the business system with the data in the offline data warehouse, correct the data, and update and synchronize it to Paimon to achieve two-way data flow, enabling real-time queries to use the latest data.

[0150] The data synchronization module supports the integration of offline data and real-time data through timed batch updates and incremental synchronization, effectively improving the depth and accuracy of real-time data analysis.

[0151] Query service layer:

[0152] The query service layer serves as the unified data query entry for business applications and supports seamless joint queries of real-time data and historical data.

[0153] Cross-warehouse query interface design: Support the business's access requirements for real-time and historical data through API and SQL interfaces.

[0154] Query optimization: Dynamically optimize the query path according to the data type (real-time or historical), and adopt partition indexing and caching strategies to improve query performance.

[0155] Real-time-offline integrated query: The query engine supports joint queries between Paimon and Hadoop, enabling the business to obtain real-time and historical data simultaneously.

[0156] The query service layer integrates the data query capabilities of Paimon and Hadoop, enabling the insurance business to access real-time data and offline data simultaneously through one interface, achieving efficient integrated queries of real-time data and historical data.

[0157] In summary, through the data synchronization system of the embodiments of the present application, data realizes the integration of real-time analysis and historical trends through the real-time processing of Paimon and the offline storage of Hadoop. Paimon can store and calculate real-time data streams in real time, while Hadoop provides larger-scale data storage and batch processing support. The two coordinate data consistency through the synchronization module. On the one hand, the data synchronization system of the present application realizes the seamless integration of real-time data and historical data, improving the real-time performance of data analysis. On the other hand, through the joint optimization of Paimon and Hadoop, it ensures the consistency of data in real-time and offline warehouses, and can reduce the development and maintenance costs of real-time and offline data warehouses, improving the scalability and operation and maintenance convenience of the data warehouse system. Further, the query service layer ultimately enables the business layer to access all data in a unified manner, thus meeting the comprehensive data analysis needs of the insurance business.

[0158] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as Figure 7 shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. Among them, the processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used for the processor to exchange information with external devices. The communication interface of the computer device is used to communicate with external terminals in a wired or wireless manner. The wireless manner can be implemented through WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it realizes a data synchronization method applied to a data synchronization system, where the data synchronization system includes a real-time data warehouse and an offline data warehouse deployed in the same database. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covering the display screen, or a button, a trackball, or a touchpad provided on the shell of the computer device, or an external keyboard, touchpad, or mouse, etc.

[0159] Those skilled in the art can understand, Figure 7The structure shown is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0160] In an exemplary embodiment, a computer device is provided, including a memory and a processor. A computer program applied to a data synchronization system is stored in the memory. When the processor executes the computer program, the steps in the data synchronization method embodiment as described above are implemented.

[0161] In an embodiment, a computer-readable storage medium is provided, on which a computer program applied to a data synchronization system is stored. The data synchronization system includes a real-time data warehouse and an offline data warehouse deployed in the same database. When the computer program is executed by a processor, the steps in the data synchronization method embodiment as described above are implemented.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant regulations.

[0163] Those of ordinary skill in the art can understand that all or part of the processes in the above-described method embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-described method embodiments. Among them, any reference to a memory, database, or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The databases involved in the embodiments provided in the present application can include at least one of relational databases and non-relational databases. Non-relational databases can include distributed databases based on blockchain, etc., without limitation. The processors involved in the embodiments provided in the present application can be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, data processing logics based on quantum computing, etc., without limitation.

[0164] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0165] The above embodiments only represent several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the patent scope of the present application. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A data synchronization method, characterized in that: The method is applied to a data synchronization system, wherein the data synchronization system includes a real-time data warehouse and an offline data warehouse deployed in the same database, and the method includes: Update the incremental data obtained from the data source to the real-time data warehouse, wherein the incremental data is the real-time data generated by the business system; Every first period, synchronizing the incremental data updated in the real-time data warehouse as historical data to the offline data warehouse; Every second period, a verification task is generated and executed, the verification task is used to obtain full data from the data source, and verify the consistency of the full data with the data stored in the offline data warehouse, obtain a verification result, and modify the data stored in the offline data warehouse according to the verification result; the first period is smaller than the second period; The correction results of the data in the offline data warehouse are synchronized to the real-time data warehouse.

2. The method according to claim 1, characterized in that The acquiring incremental data from the data source and updating the incremental data to the real-time data warehouse includes: Acquire initial real-time data generated by the business system from the data source; Preprocessing the initial real-time data to obtain preprocessed incremental data; The incremental data is updated to the ODS of the real-time data warehouse, and a snapshot of the incremental data is generated for storage; the preprocessing includes at least one of data cleaning, data deduplication, format conversion and standardization processing.

3. The method according to claim 2, characterized in that The method further comprises: When receiving a probe request, a target snapshot is determined in the real-time data warehouse, and the abnormal data is compared with the target snapshot for consistency; the target snapshot is the snapshot of the incremental data corresponding to the abnormal data, and the abnormal data is the data pointed to by the probe request and existing in the ODS of the offline data warehouse, and inconsistent with the corresponding historical data in the DWS of the offline data warehouse; wherein the data in the ODS of the offline data warehouse is updated to the DWS every third period, and the third period is greater than the first period; If they are consistent, modifying the abnormal data based on the target snapshot; If they are inconsistent, the data in the ODS of the offline data warehouse is modified based on the target snapshot.

4. The method according to claim 1, characterized in that: The method further comprises: Obtaining a total update amount, where the total update amount is a total amount of the incremental data updated to the real-time data warehouse since the last execution of the verification task; When the total update amount is greater than the target value, generating and executing a verification task; The target value is determined based on an increment of the target duration and a curve relationship, wherein the curve relationship represents a relationship between an increment of data in each of the verification tasks and an increment of the task duration used to execute the task.

5. The method according to claim 1, characterized in that The step of synchronizing the incremental data updated in the real-time data warehouse to the offline data warehouse includes: According to the list of the incremental data recorded in the target update log, the incremental data is updated into the offline data warehouse; The target update log records a list of each incremental data updated in the real-time data warehouse after the last update time, and the update time is the time when the real-time data warehouse updates the incremental data to the offline data warehouse.

6. The method according to claim 5, characterized in that The process of determining the target update log includes: Checking the first update log and the second update log, and determining a new update log as the target update log based on the corresponding non-repetitive incremental data in the first update log and the second update log; The first update log records that the data source updates the list of incremental data to the real-time data warehouse through the first channel, and the second update log records that the data source updates the list of incremental data to the real-time data warehouse through the second channel.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: When receiving a cross-time and space query request, querying the offline data warehouse and the real-time data warehouse respectively based on the cross-time and space query request; The cross-time and space query request is a query request for both the historical data and the real-time data; The query results for the real-time data warehouse and the offline data warehouse are integrated into a target query result, and the target query result is fed back to the initiator of the cross-temporal and spatial query request.

8. A data synchronization system, characterized in that: The data synchronization system includes a data synchronization module, and a real-time data warehouse and an offline data warehouse deployed in the same database, wherein: The real-time data warehouse acquires incremental data from the real-time data source to update the data; the incremental data is the real-time data generated by the business system; The data synchronization module is used to synchronize the incremental data generated in the real-time data warehouse as historical data to the offline data warehouse every first period; The data synchronization module is further used to obtain full data from the data source every second period, and check the consistency of the full data with the data stored in the offline data warehouse to obtain a check result, and modify the data stored in the offline data warehouse according to the check result; the second period is greater than the first period; The data synchronization module is also used to synchronize the correction results of the data in the offline data warehouse to the real-time data warehouse.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images