Real-time integration system, method and equipment based on life insurance customer data and medium

Abstract

The invention provides a real-time integration system, method and device based on life insurance customer data and a medium, and the system comprises a data monitoring module which is used for obtaining the change operation of customer related data in a life insurance core business system, and writing the change data into a distributed message queue; the streaming computation module is used for consuming the change data in the distributed message queue based on an Flink streaming computation engine, querying dimension table data in a distributed column database in real time, performing linkage query and push-back update on associated customer data according to preset association logic, and generating a customer panoramic view; and the real-time data write-in module is used for writing the panoramic view of the client into a corresponding record in the distributed column database and providing a uniform interface for the outside to respond to a client data query request. The problem of data inconsistency caused by task failure or order disorder is effectively avoided, and the reliability of the data integration process is improved.

Description

Technical Field

[0001] This application relates to the fields of big data processing and real-time data integration technology, and in particular to real-time integration systems, methods, devices and media based on life insurance customer data. Background Technology

[0002] In the current process of building information systems for the life insurance industry, customer data, as a core asset, is widely distributed across multiple business subsystems such as underwriting, claims, customer service, and policy maintenance. Due to the independent construction and maintenance of each system, customer data suffers from problems such as scattered sources, inconsistent formats, and chaotic identification systems, making it difficult to form a complete and consistent panoramic view of the customer base. Existing technologies generally employ an offline batch processing mode (T+1 method) based on Oracle databases for customer data integration, but this reveals significant shortcomings when facing the ever-increasing demands for real-time business: First, poor data update timeliness. Relying on daily batch processing tasks, the latest customer behavior (such as policy purchases, cancellations, and consultation records) takes at least 24 hours to be reflected in the integrated data, failing to meet the needs of application scenarios sensitive to response speed, such as real-time marketing recommendations and online customer service decision support. Second, weak ability to handle abnormal data. In the offline processing flow, if dirty data or logical conflicts are found, a skip or delayed correction strategy is usually adopted to avoid affecting the overall batch processing progress, thereby compromising data integrity. Third, a lack of linkage mechanism for data updates. When a master table (such as the customer basic information table) changes, its associated sub-tables (such as policy details, agent relationships, etc.) still need to be updated sequentially in a fixed order by the scheduling system. If an intermediate link fails, the subsequent links are interrupted, leading to inconsistent data states or even "data gaps." Furthermore, the serial update mechanism also limits the system's concurrent processing capabilities and response flexibility. Therefore, there is an urgent need to build a real-time customer data integration platform that supports high throughput, low latency, and strong consistency, breaking through the technical bottlenecks of the traditional T+1 model and realizing a shift from "passive response" to "proactive perception," providing a solid data foundation for the digital transformation of life insurance. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a real-time integration system, method, device and medium based on life insurance customer data, which realizes millisecond-level capture and processing of life insurance customer data, reducing the original T+1 daily latency to the second level or even the sub-second level. The system can automatically trigger real-time queries and back-calculation of related dimension tables. Combined with transaction control and retry mechanisms, it effectively avoids data inconsistency problems caused by task failure or disordered sequence, and improves the reliability of the data integration process.

[0004] This application provides a real-time integration system based on life insurance customer data. The real-time integration system includes a data monitoring module, a streaming computing module, and a real-time data writing module, which are sequentially connected in communication. The data monitoring module is used to acquire changes to customer-related data in the core life insurance business system and write the changed data to a distributed message queue. The streaming computing module is used to consume changed data in the distributed message queue based on the Flink streaming computing engine, query dimension table data in the distributed columnar database in real time, and perform linked queries and pushback updates on related customer data according to preset association logic to generate a panoramic view of customers; wherein, the distributed message queue is a Kafka cluster, and different business topics correspond to independent topics and consumer groups. The real-time data writing module is used to write the customer panoramic view into the corresponding record in the distributed columnar database and provide a unified interface to respond to customer data query requests.

[0005] In one possible implementation, when the data monitoring module is used to obtain changes to customer-related data in the core life insurance business system and write the changed data to a distributed message queue, the data monitoring module is specifically used for: The log files in the core business system of life insurance are parsed using DSG-type real-time data replication tools to identify the addition, deletion, and modification operations of customer basic information, policy information, and service interaction records. Based on business categories, add, delete, and modify operations on customer basic information, policy information, and service interaction records are distributed to different business topics in the Kafka cluster.

[0006] In one possible implementation, the streaming computing module is further configured to: When a data change event is detected in the business topic corresponding to the driving table, the Flink streaming computing engine automatically loads the mapping rules and triggers remote query and incremental calculation operations for at least one sub-table data according to the preset business relationship; wherein, the sub-table data includes at least one of the organization code table, product holding table and service interaction log table; By broadcasting events to multiple processing operators, state synchronization and data re-aggregation are completed, ensuring consistent updates of the customer's panoramic view.

[0007] In one possible implementation, the real-time integration system further includes a historical data processing module, which is communicatively connected to the streaming computing module; wherein, The historical data processing module is used to extract historical customer data from the big data warehouse, clean the historical customer data, convert it into a hash file, and import it in batches into the pre-partitioned distributed columnar database.

[0008] In one possible implementation, the streaming computing module is further configured to: When querying the second type of dimension table data based on the first type of change event, upon receiving the response data, it is verified in reverse whether the response data affects the state of the record corresponding to the original driving event. If there is an impact, the compensation calculation process is started and the exception log is recorded.

[0009] In one possible implementation, before writing the customer panoramic view to the corresponding record in the distributed columnar database, the real-time data writing module is further configured to: The operation flag bits in the customer panoramic view are identified by data parsing, and the distributed columnar database is partitioned based on the operation flag bits.

[0010] In one possible implementation, the streaming computing module is further configured to: Based on the Flink streaming computing engine, real-time customer data streams are grouped and processed, and customer analysis metrics are dynamically calculated for each group of real-time customer data streams to determine the analysis results. The analysis results are encapsulated into structured messages, written to downstream business topics, and synchronously updated to the external storage system.

[0011] This application also provides a real-time integration method based on life insurance customer data, the real-time integration method including: Obtain changes to customer-related data in the core life insurance business system and write the changed data to a distributed message queue; Based on the Flink streaming computing engine, the system consumes the changed data in the distributed message queue, queries the dimension table data in the distributed columnar database in real time, and performs linked queries and pushback updates on related customer data according to the preset association logic to generate a panoramic view of the customer. The distributed message queue is a Kafka cluster, and different business topics correspond to independent topics and consumer groups. The customer panoramic view is written to the corresponding record in the distributed columnar database, and a unified interface is provided to respond to customer data query requests.

[0012] This application also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, the steps of the real-time integration method based on life insurance customer data described above are performed.

[0013] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the real-time integration method based on life insurance customer data as described above.

[0014] This application provides a real-time integration system, method, device, and medium based on life insurance customer data. The real-time integration system includes a data monitoring module, a streaming computing module, and a real-time data writing module, which are sequentially connected. The data monitoring module acquires changes to customer-related data in the core life insurance business system and writes the changed data to a distributed message queue. The streaming computing module consumes the changed data from the distributed message queue using the Flink streaming computing engine, queries dimension table data in a distributed columnar database in real time, and performs linked queries and pushback updates on related customer data according to preset association logic to generate a comprehensive customer view. The distributed message queue is a Kafka cluster, and different business themes correspond to independent themes and consumer groups. The real-time data writing module writes the comprehensive customer view to the corresponding records in the distributed columnar database and provides a unified interface to respond to customer data query requests. It achieves millisecond-level capture and processing of life insurance customer data, reducing the original T+1 daily latency to the second level or even sub-second level. The system can automatically trigger real-time queries and back-calculation of related dimension tables. Combined with transaction control and retry mechanisms, it effectively avoids data inconsistency problems caused by task failure or disordered sequence, and improves the reliability of the data integration process.

[0015] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is one of the structural diagrams of a real-time integration system based on life insurance customer data provided in an embodiment of this application; Figure 2 This is a second schematic diagram of the structure of a real-time integration system based on life insurance customer data provided in an embodiment of this application; Figure 3 A flowchart illustrating a real-time integration method based on life insurance customer data provided in this application embodiment; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0018] Icons: 100 - Real-time integration system based on life insurance customer data; 110 - Data monitoring module; 120 - Streaming computing module; 130 - Real-time data writing module; 140 - Historical data processing module; 400 - Electronic device; 410 - Processor; 420 - Memory; 430 - Bus. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. Based on the embodiments of this application, every other embodiment obtained by those skilled in the art without inventive effort falls within the scope of protection of this application.

[0020] First, the applicable application scenarios of this application are introduced. This application can be applied to the fields of big data processing and real-time data integration technology.

[0021] Research has revealed that in the current process of life insurance IT development, customer data, as a core enterprise asset, is widely distributed across multiple business subsystems such as underwriting, claims, customer service, and policy maintenance. Due to the independent construction and operation of each system, customer data suffers from issues such as dispersed sources, inconsistent formats, and chaotic identification systems, making it difficult to form a complete and consistent panoramic view of the customer base. Existing technologies generally employ an offline batch processing mode (T+1 method) based on Oracle databases for customer data integration, but this reveals significant shortcomings when facing the increasing demands for real-time business: First, poor data update timeliness. Relying on daily batch processing tasks, the latest customer behavior (such as policy purchases, cancellations, and consultation records) takes at least 24 hours to be reflected in the integrated data, failing to meet the needs of application scenarios sensitive to response speed, such as real-time marketing recommendations and online customer service decision support. Second, weak ability to handle abnormal data. In the offline processing flow, if dirty data or logical conflicts are found, a skip or delayed correction strategy is usually adopted to avoid affecting the overall batch processing progress, thereby compromising data integrity. Third, a lack of linkage mechanism for data updates. When a master table (such as the customer basic information table) changes, its associated sub-tables (such as policy details) still need to be updated sequentially in a fixed order by the scheduling system. If an intermediate link fails, the subsequent links are interrupted, leading to data inconsistencies or even "data gaps." Furthermore, the serial update mechanism limits the system's concurrent processing capabilities and response flexibility. Therefore, there is an urgent need to build a real-time customer data integration platform that supports high throughput, low latency, and strong consistency, breaking through the technical bottlenecks of the traditional T+1 model and achieving a shift from "passive response" to "proactive perception," providing a solid data foundation for the digital transformation of life insurance.

[0022] Based on this, this application provides a real-time integration system based on life insurance customer data, which realizes millisecond-level capture and processing of life insurance customer data, reducing the original T+1 daily latency to the second level or even the sub-second level. The system can automatically trigger real-time queries and back-calculation of related dimension tables. Combined with transaction control and retry mechanisms, it effectively avoids data inconsistency problems caused by task failure or disordered sequence, and improves the reliability of the data integration process.

[0023] Please see Figure 1 , Figure 1 This is one of the structural diagrams of a real-time integration system based on life insurance customer data provided in an embodiment of this application. Figure 1 As shown in the figure, the real-time integration system 100 based on life insurance customer data provided in this application embodiment includes a data monitoring module 110, a streaming computing module 120, and a real-time data writing module 130, which are sequentially connected in communication.

[0024] Specifically, the data monitoring module 110 is used to acquire changes to customer-related data in the core life insurance business system and write the changed data to a distributed message queue; the streaming computing module 120 is used to consume the changed data in the distributed message queue based on the Flink streaming computing engine, query dimension table data in the distributed columnar database in real time, and perform linked queries and pushback updates on related customer data according to preset association logic to generate a customer panoramic view; wherein, the distributed message queue is a Kafka cluster, and different business topics correspond to independent topics and consumer groups; the real-time data writing module 130 is used to write the customer panoramic view to the corresponding record in the distributed columnar database and provide a unified interface to respond to customer data query requests.

[0025] Here, distributed message queues (such as Kafka) are used to aggregate existing customer data from business channels such as the underwriting system, claims platform, and customer service center in real time. Streaming ETL technology is used to clean, deduplicate, and standardize key data such as policy information, payment records, and service requests in real time, and a real-time updated panoramic view of existing life insurance customers is established based on a unified customer identification system.

[0026] Among them, the real-time data writing module 130 accesses real-time JSON data in Kafka. After parsing the data, it inserts, updates, and deletes HBase data according to different operation types. Due to the large amount of customer and policy-related data, the HBase table partition concurrency processing parameters are optimized and adjusted to ensure that real-time data can be queried in the HBase table in a timely manner.

[0027] Here, the streaming computing module 120 automatically and instantly adjusts all pre-defined related tables according to logic when data changes. Dedicated tools control error handling and retry mechanisms to prevent data synchronization interruptions due to single failures. Transaction control is used in critical update steps to avoid data inconsistencies caused by partial updates. By controlling the coupling between modules and introducing a timer mechanism, the sequential execution of linked updates is ensured, avoiding circular dependencies and high-frequency updates.

[0028] It's important to note that the core architecture of the linked query consists of: 1. Driving Table: These are the core data tables that trigger a series of subsequent data processing actions once a change occurs. Examples include: customer basic information table (containing name, ID number, contact information, etc.), agent information table, and organization code mapping table. 2. Real-time Dimension Table: Dimensional data stored in HBase, used to support fast lookups and relational completion, such as product catalogs, channel categories, and region codes. 3. Result Generation Rule Engine: The logic module embedded in the Flink stream processing program is responsible for parsing event types, determining relational paths, calling query interfaces, and assembling the final output. 4. Message Channel and Topic Routing Mechanism: In Kafka, different data streams are divided by topic to ensure independent consumption and fault isolation for each business entity, while also supporting cross-topic reverse lookup mechanisms.

[0029] In a specific implementation, the execution process of the linked query is illustrated by taking "customer contact information change" as an example, which shows how the linked query is automatically launched in the system: (1) Change capture and event queuing: The customer changes the mobile phone number in the core system, and the database generates an UPDATE log; the DSG real-time replication tool listens to the change, extracts the key fields (CustomerID, new mobile phone number), and encapsulates them into a JSON message; - the message is published to the Topic named `customer_profile_change` in Kafka. (2) Flink consumes and identifies the driving event: The Flink job listens to the Topic, and after receiving the message, it parses out the CustomerID and the operation type (update). The system determines that this is a "driving table change" and starts the linked query process. (3) Initiate the first dimension table query: The Flink program uses CustomerID as the primary key to initiate an asynchronous query request to HBase to obtain the latest snapshot data of the current customer. (4) Perform reverse backtracking query (dual-drive linkage mechanism): To prevent data loss or loop omission, the system implements a two-way linkage query strategy: use the known CustomerID to query all policies under its name (forward query), and then for each policy number (PolicyNo), check whether it still points to the customer (reverse verification) to prevent data misalignment caused by policy transfer not being synchronized. Example: If a policy has been transferred to another customer, but the original customer view has not been cleared in time, the anomaly can be identified and correction triggered by the reverse query. (5) Related extended query: query the agent information of the customer (through AgentID→HBaseagent_dim table), obtain the agency code of the agent, and further query the agency hierarchy (branch company→sub-company→business area), and load the customer's historical claims number, the summary of the most recent customer service call and other service data in a linked manner. (6): Data fusion and result generation: The data fragments from multiple dimension tables are spliced ​​and merged to form a unified customer panoramic view fragment. Conflicting data (such as the coexistence of old and new mobile phone numbers) are cleaned according to timestamp or priority rules. The results are output in a standardized JSON format and written to the downstream KafkaTopic (such as `customer_360_view_update`) for external systems to subscribe to.

[0030] In one possible implementation, when the data monitoring module 110 is used to obtain changes to customer-related data in the core life insurance business system and write the changed data to a distributed message queue, the data monitoring module 110 is specifically used for: A: Based on the DSG-type real-time data replication tool, the log files in the core business system of life insurance are parsed to determine the addition, deletion and modification operations of customer basic information, policy information and service interaction records.

[0031] Here, the DSG-type real-time data replication tool has a built-in log parsing engine that can automatically identify and parse SQL-level insert, update, and delete (INSERT, UPDATE, DELETE) operations recorded in the database transaction log, and extract key metadata information such as the affected table names, operation types, changed fields, and their old and new values. The system pre-configures a set of key business data tables to be monitored, mainly including: customer basic information tables (such as `CUST_INFO`), policy master tables (such as `POLICY_MASTER`), service interaction record tables (such as `SERVICE_LOG`), etc., ensuring that only core data streams closely related to the construction of the customer panoramic view are captured.

[0032] B: Based on business categories, add, delete, and modify operations on customer basic information, policy information, and service interaction records are distributed to different business topics in the Kafka cluster.

[0033] Here, the platform introduces a data routing and distribution mechanism based on business categories. After log parsing is complete, the system dynamically maps each data change event to a different predefined business topic in the Kafka cluster, based on the original data table to which the changed data belongs and its business semantic attributes. For example, CRUD operations on the customer basic information table (`CUST_INFO`) are categorized as "customer master data" and directed to the `topic_cust_base_change` topic in the Kafka cluster; changes to the policy master table (`POLICY_MASTER`) and related product holding information are categorized as "policy business data" and sent to the `topic_policy_change` topic; service interaction logs from the customer service system, such as service requests, complaint records, and telephone interactions, are classified as "customer service behavior data" and published to the `topic_service_interaction` topic.

[0034] It should be noted that the above topic division follows the "single responsibility principle," meaning that each KafkaTopic only carries one type of highly cohesive business data stream, thereby achieving physical isolation of data channels. This approach has multiple technical advantages: First, it allows downstream Flink stream processing programs to subscribe to specific topics on demand, avoiding interference from irrelevant data and improving computational efficiency; second, if a consumption delay or anomaly occurs in a data link, it will not affect the data processing flow of other business lines, enhancing the system's fault tolerance and stability; third, it facilitates the subsequent implementation of fine-grained access control, traffic monitoring, and quality auditing.

[0035] To ensure data integrity and consistency during transmission, all parsed change events are encapsulated in structured JSON format, including necessary fields such as timestamp, operation type, source table name, primary key, list of changed fields, and before / after image. These are asynchronously submitted to the corresponding Topic via KafkaProducer. At the same time, a message acknowledgment mechanism (acks=all) and a partition key strategy are enabled to ensure that data changes for the same customer or policy always fall into the same partition, maintaining the order of events.

[0036] In one possible implementation, the streaming computing module 120 is further configured to: a: When a data change event is detected in the business theme corresponding to the driving table, the Flink streaming computing engine automatically loads the mapping rules and triggers remote query and incremental calculation operations for at least one sub-table data according to the preset business relationship; wherein, the sub-table data includes at least one of the organization code table, product holding table and service interaction log table.

[0037] Here, when the DSG-type data replication tool writes CRUD operations on customer basic information tables (such as `CUST_INFO`), agent information tables (such as `AGENT_INFO`), or policy master tables (such as `POLICY_MASTER`) in the life insurance core system to the corresponding business topic (e.g., `topic_cust_base_change`) in the Kafka cluster, the Flink stream processing job immediately detects the arrival of this data change event and initiates the pre-defined linkage processing flow. The mapping rule loading module in the Flink job dynamically loads the matching field mapping rules, association logic configurations, and query templates based on the topic and operation type of the current message. These rules are pre-stored in the system in the form of configuration files or remote metadata services, supporting flexible expansion and hot updates, ensuring that the platform can adapt to future additions of driver-driven relationships without restarting the task. Based on the predefined business relationship graph, one or more sub-table objects affected by the change in the driving table are identified.

[0038] The sub-table data includes at least the following key dimension tables closely related to the customer profile: an organization code table (used to identify the branch office to which the customer belongs), a product holding table (recording the various insurance products currently held by the customer), and a service interaction log table (saving customer service history). For each affected sub-table, the Flink job generates a corresponding remote query request and accesses the real-time dimension table database deployed in HBase or TiDB through an asynchronous I / O mechanism to obtain the latest context data.

[0039] It should be noted that the driving table refers to the table whose real-time data in the corresponding topic drives the Flink program to perform unified queries and calculate relevant results after being consumed by Flink.

[0040] b: By broadcasting events to multiple processing operators, state synchronization and data re-aggregation are completed, ensuring consistent updates of the customer's panoramic view.

[0041] To ensure state consistency and processing efficiency during multi-source data aggregation, this solution introduces a broadcast mechanism for event distribution and state synchronization. Specifically, the original driving event is encapsulated as a broadcast stream and distributed to multiple parallel processing operators downstream. Upon receiving a broadcast event, each operator independently performs data recalculation and local aggregation operations on its assigned dimensions, based on locally cached dimension table snapshots or real-time query results. For example, one operator queries the organization code table based on the customer ID to update the customer's channel and region; another operator calls the product holding table API to reassess the customer's coverage gaps and cross-selling potential; and a third operator analyzes service interaction logs to determine customer satisfaction trends and mark high-risk service events.

[0042] After completing local incremental calculations, all operators merge their outputs into a single MergedStream, which is then integrated by the final view aggregation operator to generate a complete customer profile record containing the latest customer tags, behavioral characteristics, and value scores. This profile is then written back to a unified real-time customer view table (such as the `realtime_customer_profile` table in HBase) for downstream applications to query in real time.

[0043] For further details, please refer to Figure 2 , Figure 2 This is a second schematic diagram of the structure of a real-time integration system 100 based on life insurance customer data, provided as an embodiment of this application. Figure 2As shown, the real-time integration system 100 based on life insurance customer data also includes a historical data processing module 140, which is communicatively connected to the streaming computing module 120.

[0044] Specifically, the historical data processing module 140 is used to extract historical customer data from the big data warehouse, clean the historical customer data and convert it into a hash file, and then import it in batches into the pre-partitioned distributed columnar database.

[0045] In this specific implementation, SQL is written to extract and manipulate historical data stored in Hive, and then a shell script is used to initialize and import the processed data into the HBase database. The historical data processing module 140 cleans and processes the T-1 data in the Hive table, leveraging the superior performance of rowkey queries on HBase tables. The data is then used to generate HFile files via HBase tools, and finally, the HFile files are imported into the HBase table. To ensure query efficiency, the large HBase database is pre-partitioned, keeping the HBase table partition file size as small as possible below 10GB.

[0046] In one possible implementation, the streaming computing module 120 is further configured to: when querying the second type of dimension table data based on the first type of change event, upon receiving the response data, reversely verify whether the response data affects the state of the record corresponding to the original driving event; if there is an impact, start the compensation calculation process and record the exception log.

[0047] Here, after detecting a first-type change event (such as a change in customer basic information) and triggering a remote query to second-type dimension table data (such as agent information table, product holding table, or organization code table), the system does not immediately submit the final result. Instead, it enters a response data reverse verification stage. When the Flink job receives the query response data returned by the target dimension table, it starts a preset reverse verification rule engine to analyze whether the response data contains changes to key fields that may affect the status of the record corresponding to the original driving event. For example, if the "Agent ID" in a customer A's basic information changes (i.e., a first-type change event), the system will query the agent dimension table to obtain the channel type and performance level of the new agent. However, the response data returned by the query reveals that the new agent is currently "suspended" or their affiliated organization has been deregistered—although this information does not directly change the customer record itself, according to business rules, it is prohibited to associate a customer with an invalid agent.

[0048] It should be noted that the compensation calculation process includes, but is not limited to, the following: 1. Rollback of partial update state: Undo intermediate calculation results that have been executed but not yet persisted to prevent dirty data from being written; 2. Generation of correction event: Construct a new compensation-type data event (such as "Customer affiliation anomaly alarm" or "Automatic recovery to the previous valid agent") and inject it into an independent processing channel; 3. Execution of repair logic: Automatically call the default processing solution (such as switching to a backup agent) or notify the upstream system for manual intervention according to the configuration policy; 4. Update of customer status flag: Set a special label (such as "Pending manual review") in the customer panoramic view for subsequent business processes to identify and process.

[0049] In this application, addressing the characteristics of large volumes and complex structures in life insurance customer data, a real-time data replication tool is used to synchronize core customer information to a Kafka message queue, and a monitoring mechanism is established to ensure that changes in upstream data are captured immediately. This solution significantly reduces data processing latency, achieving millisecond-level real-time processing capabilities, effectively improving data processing efficiency and accuracy, and meeting the urgent needs of the business side for data response speed. In the system design, consumption between different topics is independent; an anomaly in the consumption of a single topic will not affect the normal operation of other data streams, thus ensuring the overall stability and reliability of the system.

[0050] In one possible implementation, before writing the customer panoramic view to the corresponding record in the distributed columnar database, the real-time data writing module 130 is further configured to: perform data parsing on the customer panoramic view to identify operation flag bits in the customer panoramic view, and perform partitioning processing on the distributed columnar database based on the operation flag bits.

[0051] Here, the system pre-defines a dedicated field as an operation flag (`op_flag`) in each record to identify the current data state or change type of the record. The operation flag is automatically filled by the upstream Flink stream processing job when generating the client view snapshot, and its values ​​include, but are not limited to: `"I"` for Insert, `"U"` for Update, `"D"` for Delete, `"S"` for Staging, and `"C"` for Compensation.

[0052] When the latest version of the customer's panoramic view data is ready to be written to the distributed columnar database, the system first starts the data parsing and flag identification module to scan and parse the batch of data line by line, extract the value of the `op_flag` field in each record, and determine the physical storage partition to which it should belong according to the predefined partitioning strategy rule set.

[0053] In one possible implementation, the streaming computing module 120 is further configured to: group real-time customer data streams based on the Flink streaming computing engine, dynamically calculate customer analysis metrics for each group of real-time customer data streams, determine the analysis results, encapsulate the analysis results into structured messages, write them to downstream business topics, and synchronously update them to an external storage system.

[0054] Data records with similar characteristics are grouped into the same logical window for independent processing. Common grouping strategies include: grouping by customer identifier (e.g., `customer_id`): used to build real-time behavioral sequences for individual customers, supporting individual-level profile updates; grouping by institution / channel (e.g., `branch_code`, `agent_id`): used to analyze the overall performance trends of branches or agent teams; grouping by product type (e.g., `product_category`): used to monitor the sales popularity and customer preference changes of different insurance product lines; and grouping by region or customer group label (e.g., `city`, `risk_level`): serving regional marketing activities or risk concentration analysis.

[0055] Here, the calculated customer analysis metrics include, but are not limited to: 1. Real-time RFM model scoring to identify high-value, active customers. 2. Behavioral preference tracking metrics: statistically analyzing recent customer visits to product categories, page dwell time, click paths, and other behavioral sequences, using weighted rules or simple collaborative filtering algorithms to infer their potential protection needs. 3. Churn tendency identification metrics: if a customer has not interacted with the system for N consecutive days and has a history of policy cancellation inquiries, a "silent warning" is triggered; the churn probability score is dynamically adjusted based on overdue payment status and negative customer service sentiment identification results. 4. Cross-selling potential assessment: based on existing product portfolios and actuarial recommendation rule base, it is determined whether the customer lacks key protection items such as critical illness or medical care, and a "product list to be recommended" is generated. The calculation process of the above metrics fully utilizes Flink's state programming capabilities (such as `ValueState`, `MapState`) and incremental aggregation functions (`ReduceFunction`, `AggregateFunction`), avoiding the need to rescan the entire historical data each time, significantly reducing computational overhead. Meanwhile, all intermediate states are persisted to a distributed state backend (such as RocksDB), supporting precise restarts after task failure recovery (Exactly-Once semantic guarantee).

[0056] This application introduces Flink's grouped stream processing mechanism and stateful computation model to automate the transformation from "massive, unordered data streams" to "ordered, interpretable, and actionable" customer analysis results, overcoming the technical bottlenecks of lagging indicator updates and slow response in traditional offline batch processing modes. It provides powerful data intelligence support, especially in key scenarios such as life insurance customer lifecycle management, precision marketing triggering, and service quality optimization.

[0057] In a specific implementation, the business scenario is "Customer A completes a new insurance policy and triggers targeted marketing recommendations": Step 1: Data Change Capture Stage: Customer A completes a critical illness insurance policy purchase on a mobile device. This transaction record is written to the `POLICY_MASTER` table in the life insurance core system. The DSG tool immediately detects this INSERT operation, extracts key fields after parsing the Redo Log, encapsulates it into a JSON format change event, and publishes it to the `topic_policy_change` topic in Kafka. Step 2: Flink Stream Processing and Grouping Calculation Stage: The Flink job consumes this event, calls `keyBy("customer_id")` to group it by customer ID, and enters a sliding window (window length 5 minutes, sliding step 30 seconds). The system recognizes this operation as a new policy and initiates the following processing flow: Query the `realtime_customer_profile` table in HBase to obtain customer A's historical profile (age 35, already has medical insurance, no critical illness coverage); dynamically calculate the RFM metric to obtain a comprehensive score. Step 3: Linked Update and Reverse Verification Phase: The system identifies the driving table, automatically triggers remote queries on sub-tables, asynchronously calls the "Product Holding Table" in HBase to confirm all currently valid policies, and queries the "Institution Code Table" to verify whether the affiliated channel is operating normally; reverse verification finds that the agent to which Customer A belongs has recently met performance targets, and the institution's status is "operating," with no abnormal feedback, so there is no need to initiate the compensation process. Step 4: Customer Panorama View Update and Result Output Phase: After all verifications pass, the Flink job packages the latest customer status into a unified structure and pushes it synchronously to the StarRocks analysis library.

[0058] This application provides a real-time integration system based on life insurance customer data. The real-time integration system includes a data monitoring module, a streaming computing module, and a real-time data writing module, which are sequentially connected. The data monitoring module is used to acquire changes to customer-related data in the core life insurance business system and write the changed data to a distributed message queue. The streaming computing module is used to consume the changed data in the distributed message queue using the Flink streaming computing engine, query dimension table data in a distributed columnar database in real time, and perform linked queries and pushback updates on related customer data according to preset association logic to generate a comprehensive customer view. The distributed message queue is a Kafka cluster, and different business themes correspond to independent themes and consumer groups. The real-time data writing module is used to write the comprehensive customer view to the corresponding records in the distributed columnar database and provide a unified interface to respond to customer data query requests. It achieves millisecond-level capture and processing of life insurance customer data, reducing the original T+1 daily latency to the second level or even sub-second level. The system can automatically trigger real-time queries and back-calculation of related dimension tables. Combined with transaction control and retry mechanisms, it effectively avoids data inconsistency problems caused by task failure or disordered sequence, and improves the reliability of the data integration process.

[0059] Please see Figure 3 , Figure 3 A flowchart illustrating a real-time integration method based on life insurance customer data provided in this application embodiment. Figure 3 As shown in the embodiments of this application, the real-time integration method includes: S301: Obtain changes to customer-related data in the life insurance core business system and write the changed data to a distributed message queue.

[0060] S302: Based on the Flink streaming computing engine, consume the changed data in the distributed message queue, query the dimension table data in the distributed columnar database in real time, and perform linked queries and pushback updates on related customer data according to the preset association logic to generate a panoramic view of customers.

[0061] In this step, the Flink streaming computing engine is used to consume the changed data in the distributed message queue, query the dimension table data in the distributed columnar database in real time, and perform linked queries and pushback updates on related customer data according to the preset association logic to generate a panoramic view of the customer.

[0062] It should be noted that the customer panorama view includes basic customer information, policy information, payment records, service interaction logs, and behavioral preference data.

[0063] Here, unlike the traditional T+1 offline batch processing mode, this application uses a streaming processing architecture built with Kafka and Flink, which can achieve data updates at the second or even millisecond level. When a customer makes a new insurance application, payment, or service request, their panoramic view can be updated synchronously in a very short time, ensuring that business personnel see the "latest status".

[0064] S203: Write the customer panoramic view into the corresponding record in the distributed columnar database and provide a unified interface to respond to customer data query requests.

[0065] This step, by efficiently writing the customer panoramic view into a distributed columnar database and providing a unified, stable, and secure external query interface, solves the problems of "data lag, slow query, and fragmented view" that exist in traditional life insurance data platforms. It truly realizes the data service capability of "one customer, one view, real-time visibility", which significantly improves the insurance company's customer service and operational decision-making.

[0066] In one possible implementation, the step of acquiring changes to customer-related data from the core life insurance business system and writing the changed data to a distributed message queue includes: The log files in the core business system of life insurance are parsed using DSG-type real-time data replication tools to identify the addition, deletion, and modification operations of customer basic information, policy information, and service interaction records. Based on business categories, add, delete, and modify operations on customer basic information, policy information, and service interaction records are distributed to different business topics in the Kafka cluster.

[0067] In one possible implementation, the Flink streaming engine consumes changed data from the distributed message queue, queries dimension table data in the distributed columnar database in real time, and performs linked queries and pushback updates on related customer data according to preset association logic to generate a comprehensive customer view, including: When a data change event is detected in the business topic corresponding to the driving table, the Flink streaming computing engine automatically loads the mapping rules and triggers remote query and incremental calculation operations for at least one sub-table data according to the preset business relationship; wherein, the sub-table data includes at least one of the organization code table, product holding table and service interaction log table; By broadcasting events to multiple processing operators, state synchronization and data re-aggregation are completed, ensuring consistent updates of the customer's panoramic view.

[0068] In one possible implementation, the real-time integration method further includes: Historical customer data is extracted from the big data warehouse, cleaned, and converted into hash files, which are then imported in batches into the pre-partitioned distributed columnar database.

[0069] In one possible implementation, the real-time integration method further includes: When querying the second type of dimension table data based on the first type of change event, upon receiving the response data, it is verified in reverse whether the response data affects the state of the record corresponding to the original driving event. If there is an impact, the compensation calculation process is started and the exception log is recorded.

[0070] In one possible implementation, before writing the customer panoramic view to the corresponding record in the distributed columnar database, the real-time integration method further includes: The operation flag bits in the customer panoramic view are identified by data parsing, and the distributed columnar database is partitioned based on the operation flag bits.

[0071] In one possible implementation, the real-time integration method further includes: Based on the Flink streaming computing engine, real-time customer data streams are grouped and processed, and customer analysis metrics are dynamically calculated for each group of real-time customer data streams to determine the analysis results. The analysis results are encapsulated into structured messages, written to downstream business topics, and synchronously updated to the external storage system.

[0072] This application provides a real-time integration method based on life insurance customer data. The method includes: acquiring change operations of customer-related data in the core life insurance business system and writing the changed data into a distributed message queue; consuming the changed data in the distributed message queue using a Flink streaming computing engine, querying dimension table data in a distributed columnar database in real time, and performing linked queries and pushback updates on related customer data according to preset association logic to generate a comprehensive customer view; wherein the distributed message queue is a Kafka cluster, and different business themes correspond to independent themes and consumer groups; writing the comprehensive customer view into the corresponding records in the distributed columnar database, and providing a unified interface to respond to customer data query requests. This method achieves millisecond-level capture and processing of life insurance customer data, reducing the original T+1 daily latency to second-level or even sub-second-level latency. The system can automatically trigger real-time queries and pushback calculations of related dimension tables. Combined with transaction control and retry mechanisms, it effectively avoids data inconsistency problems caused by task failures or out-of-order processing, improving the reliability of the data integration process.

[0073] Please see Figure 4 , Figure 4This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, and a bus 430.

[0074] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, they can perform the operations described above. Figure 3 The steps of the real-time integration method based on life insurance customer data in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0075] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the above-described actions. Figure 3 The steps of the real-time integration method based on life insurance customer data in the method embodiment shown are described in detail in the method embodiment, and will not be repeated here.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0077] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the shown or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0078] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0079] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0080] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] Finally, it should be noted that the above-described embodiments are merely specific implementations of this application, used to illustrate the technical solutions of this application, and not to limit them. The scope of protection of this application is not limited thereto. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this application. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be covered within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time integration system based on life insurance customer data, characterized in that, The real-time integration system includes a data monitoring module, a streaming computing module, and a real-time data writing module, which are sequentially connected in communication. The data monitoring module is used to acquire changes to customer-related data in the core life insurance business system and write the changed data to a distributed message queue. The streaming computing module is used to consume changed data in the distributed message queue based on the Flink streaming computing engine, query dimension table data in the distributed columnar database in real time, and perform linked queries and pushback updates on related customer data according to preset association logic to generate a panoramic view of customers; wherein, the distributed message queue is a Kafka cluster, and different business topics correspond to independent topics and consumer groups. The real-time data writing module is used to write the customer panoramic view into the corresponding record in the distributed columnar database and provide a unified interface to respond to customer data query requests.

2. The real-time integration system according to claim 1, characterized in that, When the data monitoring module is used to obtain changes to customer-related data in the core life insurance business system and write the changed data to a distributed message queue, the data monitoring module is specifically used for: The log files in the core business system of life insurance are parsed using DSG-type real-time data replication tools to identify the addition, deletion, and modification operations of customer basic information, policy information, and service interaction records. Based on business categories, add, delete, and modify operations on customer basic information, policy information, and service interaction records are distributed to different business topics in the Kafka cluster.

3. The real-time integration system according to claim 1, characterized in that, The streaming computing module is also used for: When a data change event is detected in the business theme corresponding to the driving table, the Flink streaming computing engine automatically loads the mapping rules and triggers remote query and incremental calculation operations for at least one sub-table data according to the preset business relationship; wherein, the sub-table data includes at least one of the organization code table, product holding table and service interaction log table; By broadcasting events to multiple processing operators, state synchronization and data re-aggregation are completed, ensuring consistent updates of the customer's panoramic view.

4. The real-time integration system according to claim 1, characterized in that, The real-time integration system also includes a historical data processing module, which is communicatively connected to the streaming computing module; wherein... The historical data processing module is used to extract historical customer data from the big data warehouse, clean the historical customer data, convert it into a hash file, and import it in batches into the pre-partitioned distributed columnar database.

5. The real-time integration system according to claim 1, characterized in that, The streaming computing module is also used for: When querying the second type of dimension table data based on the first type of change event, upon receiving the response data, it is verified in reverse whether the response data affects the state of the record corresponding to the original driving event. If there is an impact, the compensation calculation process is started and the exception log is recorded.

6. The real-time integration system according to claim 1, characterized in that, Before writing the customer panoramic view to the corresponding record in the distributed columnar database, the real-time data writing module is further configured to: The operation flag bits in the customer panoramic view are identified by data parsing, and the distributed columnar database is partitioned based on the operation flag bits.

7. The real-time integration system according to claim 1, characterized in that, The streaming computing module is also used for: Based on the Flink streaming computing engine, real-time customer data streams are grouped and processed, and customer analysis metrics are dynamically calculated for each group of real-time customer data streams to determine the analysis results. The analysis results are encapsulated into structured messages, written to downstream business topics, and synchronously updated to the external storage system.

8. A real-time integration method based on life insurance customer data, characterized in that, The real-time integration method is applied to the real-time integration system based on life insurance customer data as described in any one of claims 1-7, and the real-time integration method includes: Obtain changes to customer-related data in the core life insurance business system and write the changed data to a distributed message queue; Based on the Flink streaming computing engine, the system consumes the changed data in the distributed message queue, queries the dimension table data in the distributed columnar database in real time, and performs linked queries and pushback updates on related customer data according to the preset association logic to generate a panoramic view of the customer. The distributed message queue is a Kafka cluster, and different business topics correspond to independent topics and consumer groups. The customer panoramic view is written to the corresponding record in the distributed columnar database, and a unified interface is provided to respond to customer data query requests.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. The machine-readable instructions are executed by the processor to perform the steps of the real-time integration method based on life insurance customer data as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the real-time integration method based on life insurance customer data as described in claim 8.

Images