Data real-time synchronization method and device, computer equipment and storage medium

Abstract

The invention discloses a data real-time synchronization method and device, computer equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that sub-account information in a source database is captured in real time through a data capturing system and written into a message queue system through a data link; consuming the sub-account information in the message queue system through a real-time data processing engine, and writing the sub-account information into a data storage system; and mapping the sub-account information stored in the data storage system into a data warehouse to realize data real-time synchronization between the data warehouse and the source database. By means of the method, real-time synchronization of the sub-account information between the data warehouse and the source database can be achieved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a real-time data synchronization method, device, computer equipment and non-volatile computer-readable storage medium. Background Art

[0002] At present, in the insurance industry, especially in the field of property insurance, the efficiency of the monthly settlement process is crucial to the company's operations. Monthly settlement not only involves the collection and verification of financial data, but also includes key links such as expense allocation. In the traditional monthly settlement process, there are certain timeliness issues, which directly affect the accuracy and timeliness of the company's financial reports.

[0003] Taking Ping An Property & Casualty Insurance's I17 product (I17, full name IFRS17, is International Financial Reporting Standard No. 17 - Insurance Contracts, which is a specific rule for accounting measurement of insurance contracts formulated by the International Accounting Standards Board (IASB)) as an example, in the monthly settlement process in 2023, the time limit for the entire monthly settlement is T+15, that is, the monthly settlement needs to be completed on the 15th day after the end of the month. Although the monthly settlement time limit has been increased to T+7 through various efforts, the time limit for I17 cost sharing is still T+5, which was further accelerated to T+2; however, in this process of improving efficiency, many challenges are faced:

[0004] First, at the end of the month, the amount of relevant data such as sub-account lists and contract group information is concentrated, and there are frequent data modifications or additions. These data are usually synchronized to the big data platform in a T+1 manner, that is, the data can only be synchronized to the big data platform on the next day. This results in that on the monthly settlement date (such as the 1st), if the sub-account data is added or modified, the operation and maintenance personnel need to manually issue tasks to synchronize the data to the big data platform before the cost allocation can be carried out; this manual process not only increases the workload, but also greatly reduces the automation and timeliness of cost allocation, and cannot meet the company's requirements for the timeliness of the monthly settlement of I17 products;

[0005] In addition, due to the delay in data synchronization, the accuracy and real-time nature of data in the monthly closing process are affected, which may lead to delays and inaccuracies in the company's financial reports, thereby affecting the company's decision-making and risk management.

[0006] In summary, how to provide a real-time data synchronization method, apparatus, computer device, and non-volatile computer-readable storage medium that can achieve real-time synchronization of sub-account information between a data warehouse and a source database is a problem that needs to be urgently solved by those skilled in the art. Summary of the invention

[0007] In view of the above-mentioned deficiencies in the prior art, the object of the present invention is to provide a real-time data synchronization method, apparatus, computer equipment and non-volatile computer-readable storage medium that can be used in financial technology or other related fields, which can realize real-time synchronization of sub-account information between a data warehouse and a source database.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] A real-time data synchronization method, comprising:

[0010] The data capture system captures sub-account information in the source database in real time and writes it into the message queue system through the data link;

[0011] Consuming the sub-account information in the message queue system through a real-time data processing engine and writing it into a data storage system;

[0012] The sub-account information stored in the data storage system is mapped to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database.

[0013] In a further technical solution, the real-time data synchronization method, wherein after mapping the sub-account information stored in the data storage system to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database, further comprises:

[0014] Data calculation is performed based on the sub-account information in the data warehouse.

[0015] In a further technical solution, the real-time data synchronization method, wherein the sub-account information in the source database is captured in real time by the data capture system, and written into the message queue system through the data link, wherein:

[0016] The sub-account information includes the addition, modification or deletion of the sub-account list.

[0017] In a further technical solution, the real-time data synchronization method, wherein the sub-account information in the source database is captured in real time by the data capture system, and written into the message queue system through the data link, wherein:

[0018] The data capture system is an OGG system;

[0019] The source database is an Oracle database;

[0020] The message queue system is a Kafka system.

[0021] In a further technical solution, the real-time data synchronization method, wherein the sub-account information in the message queue system is consumed by a real-time data processing engine and written into a data storage system, wherein:

[0022] The real-time data processing engine is a Spark / Flink engine;

[0023] The data storage system is the Hudi system.

[0024] In a further technical solution, the real-time data synchronization method, wherein the sub-account information stored in the data storage system is mapped to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database, wherein:

[0025] The data warehouse is Hive.

[0026] In a further technical solution, the real-time data synchronization method, wherein the data calculation based on the sub-account information in the data warehouse specifically includes:

[0027] When allocating monthly expenses in I17, big data calculations are performed based on the sub-account list in the data warehouse.

[0028] A real-time data synchronization device, comprising:

[0029] The capture module is used to capture the sub-account information in the source database in real time through the data capture system and write it into the message queue system through the data link;

[0030] A consumption module, used to consume the sub-account information in the message queue system through a real-time data processing engine and write it into a data storage system;

[0031] A mapping module is used to map the sub-account information stored in the data storage system to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database.

[0032] A computer device, wherein the computer device comprises at least one processor; and

[0033] a memory communicatively connected to the at least one processor; wherein,

[0034] The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the real-time data synchronization method as described in any one of the above items can be implemented.

[0035] A non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the real-time data synchronization method as described in any one of the above items can be implemented.

[0036] Compared with the prior art, the present invention provides a method, device, computer equipment and non-volatile computer-readable storage medium for real-time data synchronization, wherein the method includes: capturing sub-account information in a source database in real time through a data capture system, and writing it into a message queue system through a data link; consuming the sub-account information in the message queue system through a real-time data processing engine, and writing it into a data storage system; mapping the sub-account information stored in the data storage system to a data warehouse, and realizing real-time data synchronization between the data warehouse and the source database. In this way, the method of the present invention can realize real-time synchronization of sub-account information between a data warehouse and a source database. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 A flowchart of a real-time data synchronization method provided by an embodiment of the present invention.

[0039] Figure 2 Another flowchart of a real-time data synchronization method provided by an embodiment of the present invention.

[0040] Figure 3 A schematic diagram of functional modules of a real-time data synchronization device provided in an embodiment of the present invention.

[0041] Figure 4 A schematic diagram of the hardware structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0043] In the description of the present invention, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps therein is not limited and can be appropriately adjusted as needed.

[0044] Various non-limiting embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0045] The applicant has found through research that, at present, in the insurance industry, especially in the field of property insurance, the efficiency of the monthly settlement process is crucial to the company's operations. Monthly settlement not only involves the aggregation and verification of financial data, but also includes key links such as cost allocation. In the traditional monthly settlement process, there are certain timeliness issues, which directly affect the accuracy and timeliness of the company's financial reports.

[0046] Taking Ping An Property & Casualty Insurance's I17 product (I17, full name IFRS17, is International Financial Reporting Standard No. 17 - Insurance Contracts, which is a specific rule for accounting measurement of insurance contracts formulated by the International Accounting Standards Board (IASB)) as an example, in the monthly settlement process in 2023, the time limit for the entire monthly settlement is T+15, that is, the monthly settlement needs to be completed on the 15th day after the end of the month. Although the monthly settlement time limit has been increased to T+7 through various efforts, the time limit for I17 cost sharing is still T+5, which was further accelerated to T+2; however, in this process of improving efficiency, many challenges are faced:

[0047] First, at the end of the month, the amount of relevant data such as sub-account lists and contract group information is concentrated, and there are frequent data modifications or additions. These data are usually synchronized to the big data platform in a T+1 manner, that is, the data can only be synchronized to the big data platform on the next day. This results in that on the monthly settlement date (such as the 1st), if the sub-account data is added or modified, the operation and maintenance personnel need to manually issue tasks to synchronize the data to the big data platform before the cost allocation can be carried out; this manual process not only increases the workload, but also greatly reduces the automation and timeliness of cost allocation, and cannot meet the company's requirements for the timeliness of the monthly settlement of I17 products;

[0048] In addition, due to the delay in data synchronization, the accuracy and real-time nature of data in the monthly closing process are affected, which may lead to delays and inaccuracies in the company's financial reports, thereby affecting the company's decision-making and risk management.

[0049] In view of the above problems, the existing technology urgently needs an improved method to achieve real-time synchronization of sub-account information data, and to improve the automation level and timeliness of I17 monthly settlement expense allocation, so as to meet the insurance industry's demand for efficient monthly settlement processes.

[0050] Therefore, to solve the above problem, please refer to Figure 1 The embodiment of the present invention provides a real-time data synchronization method, wherein the method comprises the steps of:

[0051] S100, using a data capture system to capture sub-account information in a source database in real time, and writing it into a message queue system through a data link;

[0052] S200, consuming the sub-account information in the message queue system through a real-time data processing engine and writing it into a data storage system;

[0053] S300: Map the sub-account information stored in the data storage system to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database.

[0054] In specific implementation, in this embodiment, the method of this embodiment can realize real-time synchronization of sub-account information data between the data warehouse and the source database. The following is a description of the specific implementation of this embodiment:

[0055] In step S100, the sub-account information in the source database is captured in real time by the data capture system, and written into the message queue system through the data link:

[0056] 1) Operation: Capture sub-account information in the source database in real time through the data capture system;

[0057] 2) Details: The data capture system may be Oracle GoldenGate (OGG) or other similar data capture technologies, which is responsible for monitoring changes in a source database (such as an Oracle database) and capturing addition, modification or deletion operations;

[0058] 3) Transmission: The captured sub-account information is written to a message queue system, such as Kafka, through a data link (such as a network connection or a specific data transmission protocol);

[0059] In step S200, the sub-account information in the message queue system is consumed by a real-time data processing engine, and written into a data storage system:

[0060] 1) Operation: Consume sub-account information in the message queue system through the real-time data processing engine;

[0061] 2) Details: The real-time data processing engine may be Spark or Flink, which can process high-throughput data streams and perform real-time data processing;

[0062] 3) Storage: The consumed data is written to a data storage system, such as Hudi, for further storage and management;

[0063] The step S300 maps the sub-account information stored in the data storage system to the data warehouse, so as to achieve real-time data synchronization between the data warehouse and the source database:

[0064] 1) Operation: Map the sub-account information in the data storage system to the data warehouse;

[0065] 2) Purpose: To achieve real-time data synchronization between the data warehouse and the source database to ensure data consistency and real-time performance;

[0066] 3) Details: The data warehouse can be Hive or other data warehouse solutions that support complex queries and analysis;

[0067] Detailed description:

[0068] 1) Data capture system: responsible for real-time monitoring of changes in the source database and capturing addition, modification or deletion operations;

[0069] 2) Message queue system: As the middleware for data transmission, it ensures the efficient transmission of data from the source database to the data storage system;

[0070] 3) Real-time data processing engine: processes the data received from the message queue system, performs necessary conversion and cleaning, and then writes it to the data storage system;

[0071] 4) Data storage system: provides persistent storage of data, supports efficient read and write operations and data management;

[0072] 5) Data warehouse: As the basis for data analysis and reporting, it stores mapped and converted sub-account information to support business decisions and compliance requirements;

[0073] The real-time data synchronization method provided in this embodiment is suitable for data processing scenarios that require high real-time and high reliability, especially in finance, insurance and other industries that require strict compliance with data consistency and accuracy; through this method, the consistency and real-time of data between different systems and platforms can be ensured, supporting complex data analysis and reporting requirements.

[0074] For further information, see Figure 2 , the real-time data synchronization method, wherein, after the step S300, mapping the sub-account information stored in the data storage system to the data warehouse, and realizing real-time data synchronization between the data warehouse and the source database, further comprises the steps of:

[0075] S400: Perform data calculation based on the sub-account information in the data warehouse.

[0076] Furthermore, in the real-time data synchronization method, in step S100, the sub-account information in the source database is captured in real time by a data capture system, and written into a message queue system through a data link, wherein the sub-account information includes addition, modification or deletion of a sub-account list.

[0077] Furthermore, in the real-time data synchronization method, the data calculation based on the sub-account information in the data warehouse specifically includes:

[0078] When allocating monthly expenses in I17, big data calculations are performed based on the sub-account list in the data warehouse.

[0079] In specific implementation, in this embodiment, when the I17 monthly expense apportionment is performed, big data calculations are performed based on the sub-account list in the data warehouse; that is, the method of this embodiment can specifically realize real-time synchronization of sub-account list data. Since the data warehouse is located in the big data platform, this embodiment adopts real-time synchronization of sub-account list data to the big data platform. Even if the monthly settlement date is the 1st, if there is data added, modified or deleted in the sub-account list table of the source database, the data can be written to the big data platform in real time, and the data that falls into the I17 monthly expense apportionment program is also the latest, which solves the current T+1 data synchronization problem and greatly improves the timeliness of the I17 monthly expense apportionment; that is, this embodiment can improve the automation level and timeliness of the I17 monthly expense apportionment, thereby meeting the insurance industry's demand for efficient monthly settlement processes.

[0080] The following is a description of the specific implementation of this embodiment:

[0081] The step S400, performing data calculation based on the sub-account information in the data warehouse, specifically includes: performing big data calculation based on the sub-account list in the data warehouse when allocating monthly expenses at I17;

[0082] 1) Operation: When allocating monthly expenses in I17, real-time big data calculation is performed based on the sub-account list in the data warehouse;

[0083] 2) Details: "I17" here refers to IFRS17, namely International Financial Reporting Standard No. 17 - Insurance Contracts. Monthly financial settlement and cost allocation are carried out on the basis of compliance with IFRS17 standards;

[0084] 3) Purpose: To utilize the sub-ledger list data stored in the data warehouse and use big data computing technology to process and analyze the data in real time to support accurate cost allocation and revenue recognition;

[0085] Detailed description:

[0086] Big data computing: Under the IFRS17 standard, sub-ledger list data in the data warehouse is used for real-time financial settlement and expense allocation to ensure the accuracy and compliance of financial reports.

[0087] Furthermore, in the real-time data synchronization method, in step S100, the sub-account information in the source database is captured in real time by the data capture system, and written into the message queue system through the data link, wherein:

[0088] The data capture system is an OGG system;

[0089] The source database is an Oracle database;

[0090] The message queue system is a Kafka system.

[0091] Furthermore, in the real-time data synchronization method, in step S200, the sub-account information in the message queue system is consumed by a real-time data processing engine and written into a data storage system, wherein:

[0092] The real-time data processing engine is a Spark / Flink engine;

[0093] The data storage system is the Hudi system.

[0094] Furthermore, in the real-time data synchronization method, in step S300, the sub-account information stored in the data storage system is mapped to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database, wherein:

[0095] The data warehouse is Hive.

[0096] During specific implementation, in this embodiment, the following is a detailed introduction and description of each component in this embodiment:

[0097] 1. OGG system (Oracle GoldenGate):

[0098] Introduction: Oracle GoldenGate is a platform-independent data replication software that can capture changes in the online transaction processing log (OLTP) of the database and replicate these changes to another database or data platform in real time. OGG supports data replication between heterogeneous systems to ensure data consistency and integrity;

[0099] Role in this embodiment: As a data capture system, the OGG system is responsible for capturing sub-account information changes in the Oracle database (including the addition, modification, or deletion of sub-account lists) in real time, and transmitting these changes to downstream systems in the form of logs;

[0100] 2. Oracle database:

[0101] Introduction: Oracle database is an object-relational database management system widely used in enterprise applications; it supports advanced data management functions such as transaction processing, concurrency control, data backup and recovery, etc.

[0102] Role in this embodiment: As the source database, the Oracle database stores the sub-account information generated by business operations and is the starting point of the entire data flow;

[0103] 3. Kafka system (Apache Kafka):

[0104] Introduction: Apache Kafka is a distributed stream processing platform designed for high-throughput data pipelines and stream processing; it runs in a publish-subscribe mode and supports the collection and distribution of real-time data;

[0105] Role in this embodiment: As a message queue system, the Kafka system receives data changes sent by the OGG system and provides data streams for the Spark / Flink engine;

[0106] 4. Spark Engine (Apache Spark):

[0107] Introduction: Apache Spark is an open source big data processing framework that supports batch processing and stream processing. Spark is fast, easy to use and flexible, suitable for complex data analysis tasks.

[0108] Role in this embodiment: As one of the real-time data processing engines, the Spark engine consumes data in Kafka and performs real-time processing and analysis;

[0109] 5. Flink engine (Apache Flink):

[0110] Introduction: Apache Flink is a stream-oriented processing framework and platform that supports event-driven data processing; Flink is known for its low latency and high throughput processing capabilities, suitable for real-time analysis and complex event processing;

[0111] Function in this embodiment: Similar to Spark, the Flink engine also serves as a real-time data processing engine, consumes data in Kafka, and provides stream processing capabilities;

[0112] 6. Hudi system (Apache Hudi):

[0113] Introduction: Apache Hudi is a data lake framework that aims to improve the efficiency of writing and querying data stored in the data lake; Hudi supports insert, update, and delete operations, optimizing data storage and management;

[0114] Role in this embodiment: As a data storage system, the Hudi system receives data processed by Spark / Fl ink, stores it in the data lake, and provides an efficient data reading and writing interface;

[0115] 7. Hive:

[0116] Introduction: Hive is a data warehouse software built on the Hadoop ecosystem, used to query and manage big data stored in the Hadoop file system; Hive defines a SQL-like language called HiveQL for querying data;

[0117] Role in this example: As a data warehouse, Hive stores and manages data in the Hudi system, supports complex data analysis and report generation, and provides support for business decision-making;

[0118] In this embodiment, these components together constitute an efficient and reliable data processing and analysis process. From real-time capture, transmission, processing to storage and analysis of data, each link plays an important role. Through this integrated solution, rapid data processing and in-depth insights can be achieved, meeting the company's needs for data real-time and accuracy.

[0119] It can be seen from the above method embodiments that the real-time data synchronization method provided by the present invention at least includes: capturing the sub-account information in the source database in real time through the data capture system, and writing it into the message queue system through the data link; consuming the sub-account information in the message queue system through the real-time data processing engine, and writing it into the data storage system; mapping the sub-account information stored in the data storage system to the data warehouse, and realizing the real-time data synchronization between the data warehouse and the source database; etc. In this way, the real-time synchronization of sub-account information between the data warehouse and the source database can be realized through the method of the present invention.

[0120] Specifically, in the real-time data synchronization method provided by the present invention, real-time synchronization of sub-account list data can be realized. Since the data warehouse is set in the big data platform, the present invention adopts real-time synchronization of sub-account list data to the big data platform. Even if the monthly settlement date is the 1st, the sub-account list table of the source database has data added, modified or deleted, the data can be written to the big data platform in real time, and the data that falls into the I17 monthly settlement expense allocation program is also the latest, which solves the current T+1 time limit data synchronization problem and greatly improves the timeliness of the I17 monthly settlement expense allocation; that is, the method of the present invention can improve the automation level and timeliness of the I17 monthly settlement expense allocation, thereby meeting the insurance industry's demand for efficient monthly settlement processes.

[0121] It should be understood that, although the present application provides method operation steps as described in the embodiments or flowcharts, more or less operation steps may be included based on conventional or non-creative labor, and these operation steps are not necessarily performed in sequence according to the embodiment or flowchart. The order of steps listed in the embodiment or flowchart is only one way of executing the order of many steps, and does not represent the only execution order. It should be noted that there is not necessarily a certain order between the above steps. A person of ordinary skill in the art can understand from the description of the embodiment of the present invention that in different embodiments, the above steps may have different execution orders, that is, they may be executed in parallel, or they may be executed in exchange, etc. Moreover, at least a part of the steps in the embodiment or flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but may be executed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but may be executed in turn, alternately or synchronously with other steps or at least a part of the sub-steps or stages of other steps.

[0122] Based on the above method embodiment, please refer to Figure 3 Another embodiment of the present invention further provides a real-time data synchronization device, wherein the device comprises:

[0123] The capture module 11 is used to capture the sub-account information in the source database in real time through the data capture system, and write it into the message queue system through the data link;

[0124] A consumption module 12, used to consume the sub-account information in the message queue system through a real-time data processing engine and write it into a data storage system;

[0125] The mapping module 13 is used to map the sub-account information stored in the data storage system to the data warehouse, so as to achieve real-time data synchronization between the data warehouse and the source database.

[0126] In the specific implementation, in this embodiment, the real-time synchronization of sub-account information data between the data warehouse and the source database can be achieved through the device of this embodiment. The following is a description of the specific implementation of this embodiment:

[0127] The capture module 11 is used to capture the sub-account information in the source database in real time through the data capture system, and write it into the message queue system through the data link:

[0128] 1) Operation: Capture sub-account information in the source database in real time through the data capture system;

[0129] 2) Details: The data capture system may be Oracle GoldenGate (OGG) or other similar data capture technologies, which is responsible for monitoring changes in a source database (such as an Oracle database) and capturing addition, modification or deletion operations;

[0130] 3) Transmission: The captured sub-account information is written to a message queue system, such as Kafka, through a data link (such as a network connection or a specific data transmission protocol);

[0131] The consumption module 12 is used to consume the sub-account information in the message queue system through a real-time data processing engine and write it into a data storage system:

[0132] 1) Operation: Consume sub-account information in the message queue system through the real-time data processing engine;

[0133] 2) Details: The real-time data processing engine may be Spark or Flink, which can process high-throughput data streams and perform real-time data processing;

[0134] 3) Storage: The consumed data is written to a data storage system, such as Hudi, for further storage and management;

[0135] The mapping module 13 is used to map the sub-account information stored in the data storage system to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database:

[0136] 1) Operation: Map the sub-account information in the data storage system to the data warehouse;

[0137] 2) Purpose: To achieve real-time data synchronization between the data warehouse and the source database to ensure data consistency and real-time performance;

[0138] 3) Details: The data warehouse can be Hive or other data warehouse solutions that support complex queries and analysis;

[0139] Detailed description:

[0140] 1) Data capture system: responsible for real-time monitoring of changes in the source database and capturing addition, modification or deletion operations;

[0141] 2) Message queue system: As the middleware for data transmission, it ensures the efficient transmission of data from the source database to the data storage system;

[0142] 3) Real-time data processing engine: processes the data received from the message queue system, performs necessary conversion and cleaning, and then writes it to the data storage system;

[0143] 4) Data storage system: provides persistent storage of data, supports efficient read and write operations and data management;

[0144] 5) Data warehouse: As the basis for data analysis and reporting, it stores mapped and converted sub-account information to support business decisions and compliance requirements;

[0145] The real-time data synchronization device provided in this embodiment is suitable for data processing scenarios that require high real-time and high reliability, especially in finance, insurance and other industries that require strict compliance with data consistency and accuracy; through this device, the consistency and real-time of data between different systems and platforms can be ensured, supporting complex data analysis and reporting requirements.

[0146] Furthermore, the real-time data synchronization device, wherein after mapping the sub-account information stored in the data storage system to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database, further comprises:

[0147] Data calculation is performed based on the sub-account information in the data warehouse.

[0148] Furthermore, in the real-time data synchronization device, the sub-account information in the source database is captured in real time by the data capture system, and written into the message queue system through the data link, wherein the sub-account information includes the addition, modification or deletion of the sub-account list.

[0149] Furthermore, in the real-time data synchronization device, the data calculation based on the sub-account information in the data warehouse specifically includes:

[0150] When allocating monthly expenses in I17, big data calculations are performed based on the sub-account list in the data warehouse.

[0151] In specific implementation, in this embodiment, when the I17 monthly expense apportionment is performed, big data calculations are performed based on the sub-account list in the data warehouse; that is, the device of this embodiment can specifically realize real-time synchronization of sub-account list data. Since the data warehouse is located in the big data platform, this embodiment adopts real-time synchronization of sub-account list data to the big data platform. Even if the monthly settlement date is the 1st, if there is data added, modified or deleted in the sub-account list table of the source database, the data can be written to the big data platform in real time, and the data that falls into the I17 monthly expense apportionment program is also the latest, which solves the current T+1 data synchronization problem and greatly improves the timeliness of the I17 monthly expense apportionment; that is, the automation level and timeliness of the I17 monthly expense apportionment can be improved through this embodiment, thereby meeting the insurance industry's demand for efficient monthly settlement processes.

[0152] The following is a description of the specific implementation of this embodiment:

[0153] The data calculation based on the sub-account information in the data warehouse specifically includes: when allocating monthly expenses, performing big data calculation based on the sub-account list in the data warehouse;

[0154] 1) Operation: When allocating monthly expenses in I17, real-time big data calculation is performed based on the sub-account list in the data warehouse;

[0155] 2) Details: "I17" here refers to IFRS17, namely International Financial Reporting Standard No. 17 - Insurance Contracts. Monthly financial settlement and cost allocation are carried out on the basis of compliance with IFRS17 standards;

[0156] 3) Purpose: To utilize the sub-ledger list data stored in the data warehouse and use big data computing technology to process and analyze the data in real time to support accurate cost allocation and revenue recognition;

[0157] Detailed description:

[0158] Big data computing: Under the IFRS17 standard, sub-ledger list data in the data warehouse is used for real-time financial settlement and expense allocation to ensure the accuracy and compliance of financial reports.

[0159] Furthermore, in the real-time data synchronization device, the sub-account information in the source database is captured in real time by the data capture system, and written into the message queue system through the data link, wherein:

[0160] The data capture system is an OGG system;

[0161] The source database is an Oracle database;

[0162] The message queue system is a Kafka system.

[0163] Furthermore, the real-time data synchronization device consumes the sub-account information in the message queue system through the real-time data processing engine and writes it into the data storage system, wherein:

[0164] The real-time data processing engine is a Spark / Flink engine;

[0165] The data storage system is the Hudi system.

[0166] Furthermore, the real-time data synchronization device maps the sub-account information stored in the data storage system to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database, wherein:

[0167] The data warehouse is Hive.

[0168] During specific implementation, in this embodiment, the following is a detailed introduction and description of each component in this embodiment:

[0169] 1. OGG system (Oracle GoldenGate):

[0170] Introduction: Oracle GoldenGate is a platform-independent data replication software that can capture changes in the online transaction processing log (OLTP) of the database and replicate these changes to another database or data platform in real time. OGG supports data replication between heterogeneous systems to ensure data consistency and integrity;

[0171] Role in this embodiment: As a data capture system, the OGG system is responsible for capturing sub-account information changes in the Oracle database (including the addition, modification, or deletion of sub-account lists) in real time, and transmitting these changes to downstream systems in the form of logs;

[0172] 2. Oracle database:

[0173] Introduction: Oracle database is an object-relational database management system widely used in enterprise applications; it supports advanced data management functions such as transaction processing, concurrency control, data backup and recovery, etc.

[0174] Role in this embodiment: As the source database, the Oracle database stores the sub-account information generated by business operations and is the starting point of the entire data flow;

[0175] 3. Kafka system (Apache Kafka):

[0176] Introduction: Apache Kafka is a distributed stream processing platform designed for high-throughput data pipelines and stream processing; it runs in a publish-subscribe mode and supports the collection and distribution of real-time data;

[0177] Role in this embodiment: As a message queue system, the Kafka system receives data changes sent by the OGG system and provides data streams for the Spark / Flink engine;

[0178] 4. Spark Engine (Apache Spark):

[0179] Introduction: Apache Spark is an open source big data processing framework that supports batch processing and stream processing. Spark is fast, easy to use and flexible, suitable for complex data analysis tasks.

[0180] Role in this embodiment: As one of the real-time data processing engines, the Spark engine consumes data in Kafka and performs real-time processing and analysis;

[0181] 5. Flink engine (Apache Flink):

[0182] Introduction: Apache Flink is a stream-oriented processing framework and platform that supports event-driven data processing; Flink is known for its low latency and high throughput processing capabilities, suitable for real-time analysis and complex event processing;

[0183] Function in this embodiment: Similar to Spark, the Flink engine also serves as a real-time data processing engine, consumes data in Kafka, and provides stream processing capabilities;

[0184] 6. Hudi system (Apache Hudi):

[0185] Introduction: Apache Hudi is a data lake framework that aims to improve the efficiency of writing and querying data stored in the data lake; Hudi supports insert, update, and delete operations, optimizing data storage and management;

[0186] Role in this embodiment: As a data storage system, the Hudi system receives data processed by Spark / Fl ink, stores it in the data lake, and provides an efficient data reading and writing interface;

[0187] 7. Hive:

[0188] Introduction: Hive is a data warehouse software built on the Hadoop ecosystem, used to query and manage big data stored in the Hadoop file system; Hive defines a SQL-like language called HiveQL for querying data;

[0189] Role in this example: As a data warehouse, Hive stores and manages data in the Hudi system, supports complex data analysis and report generation, and provides support for business decision-making;

[0190] In this embodiment, these components together constitute an efficient and reliable data processing and analysis process. From real-time capture, transmission, processing to storage and analysis of data, each link plays an important role. Through this integrated solution, rapid data processing and in-depth insights can be achieved, meeting the company's needs for data real-time and accuracy.

[0191] It can be seen from the above device embodiments that the real-time data synchronization device provided by the present invention at least includes: capturing the sub-account information in the source database in real time through the data capture system, and writing it into the message queue system through the data link; consuming the sub-account information in the message queue system through the real-time data processing engine, and writing it into the data storage system; mapping the sub-account information stored in the data storage system to the data warehouse, and realizing the real-time data synchronization between the data warehouse and the source database; etc. In this way, the real-time synchronization of sub-account information between the data warehouse and the source database can be realized through the device of the present invention.

[0192] Specifically, in the real-time data synchronization device provided by the present invention, real-time synchronization of sub-account list data can be realized. Since the data warehouse is arranged in the big data platform, the present invention adopts real-time synchronization of sub-account list data to the big data platform. Even if the monthly settlement date is the 1st, the sub-account list table of the source database has data added, modified or deleted, the data can be written to the big data platform in real time, and the data falling into the I17 monthly settlement expense sharing program is also the latest, which solves the current T+1 time limit data synchronization problem and greatly improves the time limit of I17 monthly settlement expense sharing; that is, the device of the present invention can improve the automation level and timeliness of I17 monthly settlement expense sharing, thereby meeting the insurance industry's demand for efficient monthly settlement processes.

[0193] Based on the above method embodiment, please refer to Figure 4 Another embodiment of the present invention further provides a computer device, wherein the computer device 10 comprises:

[0194] memory 120 and one or more processors 110, Figure 4 A processor 110 is used as an example for description. The processor 110 and the memory 120 may be connected via a communication bus or other means. Figure 4 The example of connecting via a communication bus is taken as an example.

[0195] The processor 110 is used to complete various control logics of the computer device 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, a discrete gate or transistor logic, a discrete hardware component, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP, or any other such configuration.

[0196] The memory 120 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the computer program corresponding to the real-time data synchronization method in the embodiment of the present invention. The processor 110 executes various functional applications and data processing of the computer device 10 by running the non-volatile software programs, instructions and units stored in the memory 120, that is, the real-time data synchronization method in the above method embodiment is implemented.

[0197] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device or at least one function; the data storage area may store data created according to the use of the computer device 10, etc. In addition, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely arranged relative to the processor 110, and these remote memories may be connected to the computer device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0198] One or more units are stored in the memory 120, and when executed by one or more processors 110, a real-time data synchronization method as in any of the above method embodiments can be implemented, for example, the above described method can be implemented. Figure 1 The method comprises steps S100 to S300.

[0199] Those skilled in the art will understand that Figure 4 The hardware structure diagram shown in the figure is only a schematic diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0200] Based on the above method embodiments, another embodiment of the present invention further provides a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the real-time data synchronization method in any of the above method embodiments can be implemented, for example, the above described Figure 1 The method comprises steps S100 to S300.

[0201] As an example, the non-volatile storage medium can include a read-only memory (ROM), a programmable ROM (PROM), an electrically programmable ROM (EPROM), an electrically erasable ROM (EEPROM), or a flash memory. Volatile memory can include a random access memory (RAM) as an external cache memory. By way of illustration and not limitation, RAM can be obtained in many forms such as synchronous RAM (SRAM), dynamic RAM, (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synch link DRAM (SLDRAM), and direct Rambus RAM (DRRAM). The disclosed memory components or memories of the operating environment described herein are intended to include one or more of these and / or any other suitable types of memory.

[0202] Based on the above method embodiments, another embodiment of the present invention further provides a computer program product, the computer program product includes a computer program stored on a non-volatile computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a processor, the real-time data synchronization method in any of the above method embodiments can be implemented, for example, the above described Figure 1 The method comprises steps S100 to S300.

[0203] The embodiments described above are merely illustrative, in which the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0204] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can exist in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiment.

[0205] Conditional language such as "can," "can," "might," or "may," unless specifically stated otherwise or otherwise understood within the context as used, is generally intended to convey, among other things, that a particular embodiment can include (while other embodiments do not) a particular feature, element, and / or operation. Thus, such conditional language is also generally intended to imply that a feature, element, and / or operation is required for one or more embodiments anyway or that one or more embodiments must include logic for determining, with or without input or prompting, whether such features, elements, and / or operations are included or will be performed in any particular embodiment.

[0206] What has been described herein, in this specification and in the accompanying drawings includes examples that can provide a method, device, computer equipment, and non-volatile computer-readable storage medium for real-time synchronization of data. Of course, it is not possible to describe every conceivable combination of elements and / or methods for the purpose of describing the various features of the present disclosure, but it can be recognized that many other combinations and permutations of the disclosed features are possible. Therefore, it is obvious that various modifications can be made to the present disclosure without departing from the scope or spirit of the present disclosure, but all these various modifications should belong to the scope of protection of the claims attached to the present invention. In addition, or in an alternative, other embodiments of the present disclosure may be obvious from consideration of the present specification and the accompanying drawings and from the practice of the present disclosure as presented herein. It is intended that the examples proposed in the present specification and the accompanying drawings are considered to be illustrative and not restrictive in all respects. Although specific terms are used herein, they are used in a general and descriptive sense and are not used for limiting purposes.

Claims

1. A real-time data synchronization method, It is characterized in that include: The data capture system captures sub-account information in the source database in real time and writes it into the message queue system through the data link; Consuming the sub-account information in the message queue system through a real-time data processing engine and writing it into a data storage system; The sub-account information stored in the data storage system is mapped to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database.

2. The real-time data synchronization method according to claim 1, It is characterized in that After mapping the sub-account information stored in the data storage system to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database, the method further includes: Data calculation is performed based on the sub-account information in the data warehouse.

3. The real-time data synchronization method according to claim 2, It is characterized in that The data capture system captures the sub-account information in the source database in real time and writes it into the message queue system through the data link, wherein: The sub-account information includes the addition, modification or deletion of the sub-account list.

4. The real-time data synchronization method according to claim 3, It is characterized in that The data capture system captures the sub-account information in the source database in real time and writes it into the message queue system through the data link, wherein: The data capture system is an OGG system; The source database is an Oracle database; The message queue system is a Kafka system.

5. The real-time data synchronization method according to claim 4, It is characterized in that The sub-account information in the message queue system is consumed by the real-time data processing engine and written into the data storage system, wherein: The real-time data processing engine is a Spark / Flink engine; The data storage system is the Hudi system.

6. The real-time data synchronization method according to claim 5, It is characterized in that The sub-account information stored in the data storage system is mapped to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database, wherein: The data warehouse is Hive.

7. The real-time data synchronization method according to claim 6, It is characterized in that The performing data calculation based on the sub-account information in the data warehouse specifically includes: When allocating monthly expenses in I17, big data calculations are performed based on the sub-account list in the data warehouse.

8. A real-time data synchronization device, It is characterized in that include: The capture module is used to capture the sub-account information in the source database in real time through the data capture system and write it into the message queue system through the data link; A consumption module, used to consume the sub-account information in the message queue system through a real-time data processing engine and write it into a data storage system; A mapping module is used to map the sub-account information stored in the data storage system to the data warehouse to achieve real-time data synchronization between the data warehouse and the source database.

9. A computer device, It is characterized in that The computer device comprises at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor. When the computer program is executed by the at least one processor, the real-time data synchronization method according to any one of claims 1 to 7 can be implemented.

10. A non-volatile computer-readable storage medium, It is characterized in that The non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by at least one processor, the real-time data synchronization method according to any one of claims 1 to 7 can be implemented.

Images