Insurance industry distributed data processing method and device

Abstract

The invention discloses an insurance industry distributed data processing method and device, and the method can effectively solve the problems of performance bottleneck and poor expansibility of a traditional centralized database in a mass data scene in the insurance industry. Stream batch integrated processing of high-concurrency transaction processing and batch analysis tasks is realized, and the system processing efficiency and availability are improved.

Description

Technical Field

[0001] This invention relates to the field of computer software / information technology, and in particular to a distributed data processing method and apparatus for the insurance industry. Background Technology

[0002] With the widespread adoption of the internet, the continuous development of the insurance industry has led to a surge in data and business volume. Existing monolithic systems have gradually become unable to handle the increased pressure, prompting companies to begin splitting their business systems. Simultaneously, with digital transformation, the value of business data is increasing. When upgrading their technologies, companies are using distributed technologies to address the storage and performance bottlenecks of monolithic services, such as distributed microservices and distributed databases; and they are using streaming computing and big data technologies to solve data processing and massive data analysis problems, such as Flink and Hadoop. However, a satisfactory solution remains elusive for addressing the data fragmentation caused by the splitting of monolithic systems and for better utilizing data from multiple systems.

[0003] The widespread adoption of internet applications has led to a continuous increase in data volume. However, once databases reach a certain scale, their performance gradually declines, with storage and I / O becoming prominent bottlenecks for monolithic databases. Oracle RAC clusters have enabled Oracle to evolve from a single-machine model to a multi-machine parallel model, achieving load balancing and failover across database nodes and ensuring high application availability. However, compared to single-machine systems, RAC clusters have more complex underlying technologies and are more difficult to manage. Because the cluster shares resources, there is a risk of resource contention, especially in large-scale data and high-concurrency scenarios. Furthermore, their scalability is poor, leading to system and performance bottlenecks.

[0004] In the insurance industry, Oracle RAC databases are typically used to support transaction and batch processing operations. Data processing is handled by stored procedures. As business data volumes grow, single tables can now contain hundreds of millions of records, pushing database storage capacity to the terabyte level. During data processing, multi-table joins frequently cause database server bottlenecks, such as the typical merge operation (incremental merging of full records). Under a RAC architecture, the time consumed by merging tables with hundreds of millions of records is considerable, necessitating table partitioning and cache optimization – both time-consuming and labor-intensive. Furthermore, recent technological bottlenecks have occasionally occurred, making domestic production and technological self-sufficiency a major challenge and a breakthrough for the insurance industry. Therefore, based on the above architectural characteristics, actual business needs, and technical requirements, a new architecture is urgently needed to replace it. Summary of the Invention

[0005] One objective of this invention is to propose a distributed data processing method for the insurance industry. Addressing numerous challenges in insurance business, such as long transaction times, uneven pressure distribution, a surge in online business, a rapid increase in data volume, high business complexity, and numerous batch processing tasks, this method utilizes the open-source, domestically developed centralized database openGauss and incorporates PDFC-Sharding, a lightweight data middleware developed by PICC Group. This constructs a distributed data stream batch processing architecture consisting of a database middleware, a centralized database, a data replication tool, stream processing, and MPP.

[0006] Another objective of this invention is to provide a distributed data processing device for the insurance industry.

[0007] To achieve the above objectives, a first aspect of the present invention provides a distributed data processing method for the insurance industry, comprising: S1 builds a distributed database middleware based on a domestic centralized database. Through an intelligent routing mechanism, business data is sharded to multiple database nodes according to preset rules to achieve vertical database sharding and load balancing. S2, capture real-time changes of OLTP business data in the distributed database, convert the changed data into a standardized data format and transmit it to the message queue to form a heterogeneous data synchronization channel; S3, distribute the data stream in the message queue to the MPP architecture database cluster, generate a distributed execution plan based on SQL parsing and push it down to the corresponding nodes for parallel computing; S4 monitors the status of database nodes through the federated cluster management component. When the primary node fails, it automatically switches to the backup node and feeds back the batch processing results to the front-end application.

[0008] In one embodiment of the present invention, the construction of a distributed database middleware based on a domestically produced centralized database, which uses an intelligent routing mechanism to shard business data to multiple database nodes according to preset rules to achieve vertical database sharding and load balancing, further includes: S11, by adding the ShardingKey: ShardingValue identifier to the request header of the application system, the PDFC-Sharding middleware parses the identifier and uses the HASH algorithm to determine the target database sharding cluster; S12, configure the targetServerType parameter in the JDBC connection string to master or slave, and combine it with the autoBalance parameter to realize automatic load balancing and failover of master and slave nodes.

[0009] In one embodiment of the present invention, the step of capturing real-time changes of OLTP business data in the distributed database, converting the changed data into a standardized data format and transmitting it to a message queue to form a heterogeneous data synchronization channel further includes: S21, use the exBase tool to capture the underlying logs of the openGauss database in real time, and generate incremental data change records by parsing the DML operations in the binlog logs; S22, the incremental data is serialized using JSON format, and asynchronous transmission of heterogeneous data across systems is achieved through Kafka message queue.

[0010] In one embodiment of the present invention, the step of distributing the data stream in the message queue to the MPP architecture database cluster, generating a distributed execution plan based on SQL parsing, and pushing it down to the corresponding nodes for parallel computing further includes: S31 reads JSON data streams from the message queue via Kafka stream and converts them into a columnar storage format that can be processed by the MPP architecture. S32, based on the SQL parsing results, splits the computation task into multiple subtasks, and pushes the subtasks down to the MPP node containing the corresponding data partition according to the data distribution characteristics.

[0011] In one embodiment of the present invention, the step of monitoring the database node status through the federated cluster management component, automatically switching to the backup node when the primary node fails, and simultaneously feeding back the batch processing results to the front-end application further includes: S41 utilizes the CM cluster management component built into openGauss to monitor the CPU utilization, memory usage, and disk I / O throughput of each node in real time. S42, when a primary node failure is detected, the optimal node is selected from the backup nodes for primary / backup switching according to the load balancing strategy configured by the autoBalance parameter.

[0012] In one embodiment of the present invention, it further includes: S5 performs data verification operations on the data stream transmitted to the MPP architecture. By comparing the differences between the primary key hash values ​​of the source database and the target MPP node, a compensation mechanism is used to retransmit lost or abnormal data.

[0013] To achieve the above objectives, a second aspect of the present invention provides a distributed data processing device for the insurance industry, comprising: The distributed database middleware building module is used to build distributed database middleware based on domestic centralized databases. It uses an intelligent routing mechanism to shard business data to multiple database nodes according to preset rules, thereby achieving vertical database sharding and load balancing. The real-time data capture and synchronization module is used to capture real-time changes in OLTP business data in a distributed database, convert the changed data into a standardized data format and transmit it to a message queue, forming a heterogeneous data synchronization channel. The data stream distribution and parallel computing module is used to distribute the data stream in the message queue to the MPP architecture database cluster, generate a distributed execution plan based on SQL parsing and push it down to the corresponding nodes for parallel computing; The federated cluster monitoring and failover module is used to monitor the status of database nodes through the federated cluster management component. When the primary node fails, it automatically switches to the backup node and feeds back the batch processing results to the front-end application.

[0014] The methods and apparatus of this invention can effectively solve the performance bottlenecks and scalability problems of traditional centralized databases in the insurance industry under massive data scenarios. By combining distributed architecture with MPP, it can achieve high-concurrency, low-latency integrated stream and batch data processing, thereby improving system availability and resource utilization. Attached Figure Description

[0015] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a distributed data processing method for the insurance industry provided in this embodiment of the invention; Figure 2 An overall architecture diagram of another distributed data processing method for the insurance industry provided by an embodiment of the present invention; Figure 3 This is a distributed high-availability cluster design diagram provided in an embodiment of the present invention; Figure 4 This is a diagram illustrating the streaming data processing framework provided in an embodiment of the present invention. Figure 5 This is a structural diagram of a distributed data processing device for the insurance industry provided in an embodiment of the present invention. Detailed Implementation

[0016] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0017] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0018] The following are the technical terms that may be used in this invention: Distributed technology: Distributed technology typically refers to distributed data computing technology. With the widespread adoption of the internet, the volume of data has exploded, reaching petabyte (PB) levels. Traditional centralized data processing is gradually becoming inadequate for current needs, leading to the emergence of distributed technology. This involves distributing large amounts of data across multiple computer nodes for processing. In contrast to traditional centralized data processing, distributed processing distributes data across different processing units, enabling parallel processing. Therefore, distributed technology can fully utilize the computing and storage capabilities of computer clusters, improving data processing speed and increasing the volume of data processed.

[0019] Distributed databases are a product of the combination of distributed technology and databases. Their core technologies include data sharding, data synchronization, data consistency, data querying, and transaction processing. Data sharding, or distributed technology, disperses data across multiple computing nodes, enabling parallel processing and load balancing. Data synchronization technology ensures data consistency across nodes by creating data replicas, ensuring high availability when one node fails and other nodes can take over the data and provide services. Data querying and transaction processing allow users to perform data queries and transactions across multiple nodes in a collaborative environment. Distributed query engines and distributed transaction managers ensure query performance and transaction integrity and consistency. Furthermore, distributed databases can be dynamically scaled according to actual needs, addressing issues such as decreased retrieval performance due to excessively large single-table data volumes and I / O bottlenecks in single-machine databases.

[0020] Message queues: Message queues are components used for asynchronous communication between applications or different components. They are used for message storage and forwarding. Message producers send messages to message queues for storage, and downstream consumers consume the messages, realizing asynchronous message transmission, such as Kafka and RocketMQ. Through asynchronous message forwarding, system decoupling and flow control are achieved.

[0021] Stream computing: Stream computing, also known as real-time computing, is a computing model that performs real-time analysis and processing on continuous data streams. It features real-time performance, high efficiency, and scalability.

[0022] Batch processing: Batch processing, or batch computing, is different from streaming computing. Batch processing is a computing mode that analyzes and processes data within a certain time period. Generally, batch processing requires a large amount of data to be processed, has more complex business logic, and consumes more resources.

[0023] MPP (Massively Parallel Processing) is a share-nothing database architecture concept, referring to a large-scale system composed of multiple physically independent nodes interconnected via a high-speed network, working collaboratively. MPP distributes tasks in parallel across multiple servers and nodes. After computation is completed on each node, the results are aggregated to obtain the final result. Each node in an MPP database system has its own resources, such as disk, memory, and CPU. Each node manages its own data, working collaboratively and processing in parallel, effectively improving data processing efficiency.

[0024] The following description, with reference to the accompanying drawings, describes a distributed data processing method and apparatus for the insurance industry according to an embodiment of the present invention.

[0025] Example 1 This embodiment provides a distributed data processing method for the insurance industry. For example... Figure 1 As shown, the method includes the following steps: S1 is a distributed database middleware built on a domestically produced centralized database. It uses an intelligent routing mechanism to shard business data to multiple database nodes according to preset rules, thereby achieving vertical database sharding and load balancing.

[0026] Specifically, in some implementations, building a distributed database middleware based on a domestically developed centralized database involves introducing an intelligent routing mechanism to distribute business data to multiple database nodes according to preset sharding rules, thereby achieving vertical database sharding and load balancing. The technical implementation principle of this step is based on the database middleware PDFC-Sharding, whose core function is to parse upper-layer business requests and route them to the corresponding database instances according to the sharding strategy, thus achieving distributed data storage and access.

[0027] PDFC-Sharding integrates openGauss's JDBC driver (e.g., `opengauss-jdbc`, version 3.0.3) to insert middleware logic between the application layer and the database. Before an application request is sent to the database, it first passes through the middleware's routing module. This module extracts `ShardingKey: ShardingValue` from the request header and calculates the target database node according to preset sharding rules (e.g., HASH, range sharding). For example, when using a HASH algorithm, the middleware maps `ShardingValue` to a shard identifier using a hash function, and then forwards the request to the corresponding database instance based on that identifier. The connection string specifies the master / slave node using the `targetServerType=master` or `targetServerType=slave` parameter, achieving read / write separation and high availability failover.

[0028] This middleware supports configuring connection information for multiple database nodes, such as `jdbc:opengauss: / / 10.57.0.49:5432,10.57.0.50:5432,10.57.0.51:5432 / poc_robot_db`, and supports automatic load balancing (`autoBalance=true`). Sharding strategies can be configured as `HASH`, `MOD`, or `RANGE`, and sharding fields support various data types, such as integers and strings. Furthermore, the middleware supports transaction consistency control, ensuring ACID properties are maintained during multi-node writes.

[0029] This step is applicable to core transaction systems in the insurance industry, such as policy management and claims processing modules. By sharding business data according to fields such as user ID and policy number, the storage and I / O pressure on a single-node database can be effectively alleviated, improving the overall system throughput and response speed. Simultaneously, combined with openGauss's CM cluster management component, automatic failover between primary and backup nodes can be achieved, ensuring business continuity.

[0030] This step addresses the performance bottleneck of traditional centralized databases in high-concurrency, high-volume scenarios by introducing middleware to achieve intelligent data sharding and routing. Simultaneously, a vertical database partitioning strategy isolates and stores data from different business modules, improving system maintainability and scalability. The load balancing mechanism further optimizes database resource utilization and reduces the risk of single-node overload, thereby significantly improving the stability and processing efficiency of the insurance business system.

[0031] Furthermore, S1 includes: S11, by adding the ShardingKey: ShardingValue identifier to the request header of the application system, the PDFC-Sharding middleware parses the identifier and uses a HASH algorithm to determine the target database sharding cluster.

[0032] Specifically, in some implementations, this method adds a `ShardingKey: ShardingValue` identifier to the application system's request header. The self-developed middleware PDFC-Sharding parses this identifier and uses a hash algorithm to determine the target database sharding cluster, thereby achieving request routing and load balancing for the distributed database. This step is a crucial component of the entire distributed data processing architecture, and its technical implementation is based on the HTTP request header extension mechanism and the middleware's routing strategy.

[0033] When an application system initiates a database access request, it needs to explicitly add the `ShardingKey` field to the HTTP request header, with the value `ShardingValue`. This value is typically a field with unique or shardable characteristics in the business logic, such as a user ID or policy number. The PDFC-Sharding middleware, acting as a front-end proxy layer for database access, is responsible for intercepting the request and extracting this field. After extraction, the middleware makes routing decisions based on a preset sharding strategy (such as HASH, range, tag, etc.). In this solution, a HASH algorithm is used. Its core idea is to map `ShardingValue` to an integer value using a hash function, and then use this value modulo the number of sharded clusters to determine the target database node. For example, if the number of sharded clusters is... The target database partition number is .

[0034] The selection of `ShardingKey` must meet requirements such as high cardinality, low repetition rate, and business irrelevance to ensure uniform data distribution. The hash function can be consistent hashing or standard hashing (such as MurmurHash, SHA-1 truncation, etc.) to support dynamic scaling. The middleware configuration must clearly define the address list of the sharded cluster, connection string parameters (such as `targetServerType=master`, `autoBalance=true`), and the implementation method of the sharding strategy.

[0035] This step is widely used in core transaction systems, policy management systems, and claims processing systems in the insurance industry, and is especially suitable for high-concurrency, high-data-volume business scenarios. By adding a sharding identifier to the request header, the system can route requests from different users to different database nodes, achieving vertical database sharding. This effectively alleviates the storage and computing pressure on a single node, improving the overall throughput and response speed of the system.

[0036] This step enables intelligent routing of database access requests, avoiding the performance bottlenecks of traditional centralized databases. Simultaneously, high availability mechanisms in the middleware (such as master-slave failover) ensure system stability. Combined with openGauss's JDBC driver and cluster management component CM, this solution provides flexible, scalable, and high-performance database access control capabilities in the context of domestic substitution, serving as a crucial foundation for building distributed data processing architectures.

[0037] S12, configure the targetServerType parameter in the JDBC connection string to master or slave, and combine it with the autoBalance parameter to realize automatic load balancing and failover of master and slave nodes.

[0038] Specifically, in some implementations, configuring the `targetServerType` parameter in the JDBC connection string to `master` or `slave`, combined with the `autoBalance` parameter, is one of the key steps in achieving automatic load balancing and failover between the primary and secondary nodes of the database. This step relies on the connection parameter mechanism provided by the openGauss database JDBC driver (such as `opengauss-jdbc` version 3.0.3), which enables intelligent routing and switching between the primary and secondary nodes in the database cluster by specifying `targetServerType` and `autoBalance` in the connection string.

[0039] Specifically, the `targetServerType` parameter specifies the type of the target node for the connection, with options of `master` (master node) or `slave` (slave node). When set to `master`, the JDBC driver will prioritize connecting to the master node, suitable for write operations or business scenarios requiring strong consistency; when set to `slave`, the driver will connect to the slave node, suitable for read operations, thus achieving read-write separation and improving the overall system throughput. The `autoBalance` parameter controls whether automatic load balancing is enabled. When set to `true`, the driver will dynamically select the connection target based on the current node's load status, avoiding single-point overload and improving resource utilization.

[0040] This configuration is typically used in distributed database architectures in the insurance industry, as shown in the high-availability cluster design in Figure 2. By setting `jdbc:opengauss: / / 10.57.0.49:5432,10.57.0.50:5432,10.57.0.51:5432 / poc_robot_db?targetServerType=master&autoBalance=true` in the connection string, the system can achieve intelligent connection management between multiple database nodes. When the master node fails, openGauss's cluster management component CM will detect the anomaly and promote the standby node to the master node. The JDBC driver will then automatically route subsequent requests to the new master node, thereby achieving seamless failover.

[0041] Furthermore, this step relies on the JDBC driver's ability to perceive cluster topology and its intelligent scheduling mechanism for the connection pool at the technical implementation level. In the connection pool configuration, parameters such as the maximum number of connections, the minimum number of idle connections, and the connection timeout can be set, such as `maxPoolSize=50`, `minIdle=10`, and `connectionTimeout=30000`, to ensure connection stability and response speed under high concurrency scenarios.

[0042] This technical solution has significant value in the core transaction systems of the insurance industry, especially during peak business periods or node failures, effectively ensuring continuous system availability and data consistency. By properly configuring `targetServerType` and `autoBalance`, the system not only improves the reliability of database access but also reduces operational complexity and enhances the overall architecture's resilience and scalability.

[0043] S2, capture real-time changes of OLTP business data in the distributed database, convert the changed data into a standardized data format and transmit it to the message queue to form a heterogeneous data synchronization channel.

[0044] Specifically, in this application proposal, the step of "capturing real-time changes of OLTP business data in the distributed database, converting the changed data into a standardized data format and transmitting it to a message queue to form a heterogeneous data synchronization channel" is a key step in realizing the data stream batch processing architecture. This step, through underlying log capture and parsing technology, combined with lightweight data middleware and message queues, constructs an efficient, real-time data synchronization channel from OLTP to OLAP.

[0045] This step primarily relies on the database's logical log capture mechanism. It monitors the openGauss database's transaction logs (such as the Write-Ahead Log, WAL) to capture data change events occurring in OLTP operations, including INSERT, UPDATE, and DELETE operations. The capture tool exBase subscribes to the database's change logs, extracts changed data in real time, and converts it into JSON format to achieve standardized data transmission across systems and platforms. This process supports incremental synchronization, meaning it only synchronizes data that has changed since the last synchronization point, significantly reducing data transmission bandwidth and processing overhead.

[0046] exBase supports configuring capture frequency (e.g., maximum number of capture events per second), data transformation rules (e.g., field mapping, type conversion), message compression methods (e.g., GZIP, Snappy), and Kafka producer parameters (e.g., batch.size, linger.ms, compression.type). For example, batch.size can be set to 16384 bytes and linger.ms to 100 to balance throughput and latency. Additionally, the number of partitions (num.partitions) and replication factor (replication.factor) of the Kafka message queue can be configured according to business load and high availability requirements, such as setting num.partitions=12 and replication.factor=3 to ensure reliable data transmission and load balancing.

[0047] This step is widely applicable to data synchronization between core transaction systems and real-time analytics systems in the insurance industry. For example, in OLTP operations such as policy changes and claims processing, the system needs to synchronize changed data to an OLAP database with an MPP architecture in real time for analysis scenarios such as real-time report generation, risk warning, and customer profiling. The asynchronous transmission mechanism of Kafka message queues effectively decouples OLTP and OLAP systems, preventing transaction performance from being affected by data processing delays.

[0048] This step enables real-time data synchronization between heterogeneous systems, resolving the performance bottleneck of traditional databases in high-concurrency, high-volume data scenarios. By combining a standardized data format (JSON) with a message queue, not only is the reliability and consistency of data transmission improved, but a high-quality data input source is also provided for subsequent streaming processing and batch analysis, thereby significantly enhancing the real-time response capability and data processing efficiency of the insurance business system.

[0049] Furthermore, S2 includes: S21. Use the exBase tool to capture the underlying logs of the openGauss database in real time, and generate incremental data change records by parsing the DML operations in the binlog logs.

[0050] Specifically, in this technical solution, the step of "using the exBase tool to capture the underlying logs of the openGauss database in real time and generating incremental data change records by parsing the DML operations in the binlog logs" is a key link in realizing the distributed data stream batch processing architecture. Its technical implementation principle is based on the underlying log mechanism of the database and the efficient parsing capability of the data synchronization tool.

[0051] In some implementations, the exBase tool monitors the binlog stream of the openGauss database to capture DML (Data Manipulation Language) operations occurring in the database in real time, including INSERT, UPDATE, and DELETE. These operations are typically written to the binlog by the business system when executing transactions to record the before and after data changes. exBase reads the binlog logs through the database replication channel with low latency and low resource consumption, and parses them into structured JSON format data for subsequent consumption and processing by streaming frameworks (such as Kafka Streams).

[0052] Furthermore, the configuration of exBase in this step must meet certain technical specifications. For example, its maximum supported binlog read latency should be less than 100ms to ensure the real-time nature of data changes; simultaneously, its throughput should reach at least 10,000 records per second to cope with the data pressure in high-frequency trading scenarios in the insurance industry. During the parsing process, exBase extracts the table name, primary key, changed fields, and their values ​​before and after the change, based on the DML operation type, forming incremental data records. For example, for an UPDATE operation, exBase generates a JSON object containing old_value and new_value for downstream systems to perform difference processing.

[0053] This step is primarily used in OLTP and OLAP business separation architectures within the insurance industry. In actual deployments, exBase is typically deployed on a separate server in the same data center or region as the openGauss database to reduce network latency. The captured incremental data is transmitted between heterogeneous systems via Kafka message queues and is ultimately processed in batches or analyzed in real time by an MPP architecture (such as ClickHouse or GBase).

[0054] The technical advantage of this step lies in achieving low-latency, high-precision synchronization of database changes through real-time capture and parsing of binlog logs, thus providing a reliable data source for subsequent streaming processing and batch analysis. Compared to traditional synchronization methods based on triggers or scheduled tasks, this method avoids interference with database performance while improving the accuracy and timeliness of data synchronization, making it a crucial support for building a distributed data processing architecture for the insurance industry.

[0055] S22, the incremental data is serialized using JSON format, and asynchronous transmission of heterogeneous data across systems is achieved through Kafka message queue.

[0056] Specifically, in this application proposal, the step of "serializing the incremental data using JSON format and realizing asynchronous transmission of heterogeneous data across systems through Kafka message queue" is a key link in building a distributed data stream batch processing architecture. Its technical implementation principle is based on the capture and parsing mechanism of the underlying database logs, combined with the lightweight data middleware PDFC-Sharding and the streaming processing framework Kafka Stream, to achieve efficient decoupling and asynchronous transmission of OLTP and OLAP business data.

[0057] This step begins by using the data synchronization tool exBase to monitor the data tables configured for analytical applications in the openGauss database in real time. When data changes (such as INSERT, UPDATE, and DELETE operations), exBase extracts the incremental data by parsing the database's WAL (Write-Ahead Log) and serializes it into JSON format. The JSON format was chosen based on its structured nature, high readability, and good cross-platform compatibility, facilitating subsequent parsing and processing in heterogeneous systems. During serialization, data fields must maintain consistency with the schema defined in the target system to ensure correct data structure mapping.

[0058] JSON serialization must adhere to certain schema specifications, such as field type mapping, field naming rules, timestamp format (e.g., ISO 8601 standard), and primary key identifier. Kafka message queue configuration parameters include: message compression method (e.g., snappy, gzip), partitioning strategy (e.g., key-based partitioning), replication factor (replication.factor ≥ 3 to ensure high availability), and message retention strategy (retention.bytes or retention.ms). In some implementations, the message serializer can optionally be set to `org.apache.kafka.common.serialization.StringSerializer`, and deserialization can be performed using `org.apache.kafka.common.serialization.StringDeserializer` to adapt for JSON formatted data transmission.

[0059] This step is widely used in core business systems within the insurance industry, such as policy changes, claims data updates, and customer information synchronization. By asynchronously transmitting incremental data to Kafka, downstream MPP architecture systems (such as ClickHouse and GBase) can consume and process the data in real time based on Kafka Streams, thereby enabling rapid response to analytical tasks. This mechanism effectively decouples OLTP and OLAP systems, avoiding performance bottlenecks and resource contention issues caused by direct database connections.

[0060] This step enables cross-system and cross-platform heterogeneous data synchronization, ensuring data real-time performance and consistency. Through asynchronous transmission mechanisms, the system possesses excellent scalability and fault tolerance, capable of handling high-concurrency, large-volume business scenarios. Simultaneously, this step provides standardized, structured data input for subsequent streaming processing and batch analysis, serving as a core bridge for data "flow" within the entire distributed data processing architecture.

[0061] S3, distribute the data stream in the message queue to the MPP architecture database cluster, generate a distributed execution plan based on SQL parsing and push it down to the corresponding nodes for parallel computing.

[0062] Specifically, in some implementations, distributing the data stream from the message queue to the MPP architecture database cluster, and generating a distributed execution plan based on SQL parsing and pushing it down to the corresponding nodes for parallel computation is a key step in achieving efficient stream and batch processing in this solution. This step, by building a bridge between the data stream and the MPP architecture, enables unified scheduling of real-time analysis and batch processing of massive amounts of data.

[0063] Real-time data streams stored in message queues (such as Kafka) are first consumed and parsed by stream processing engines (such as Kafka Streams). The parsed data is then distributed to various compute nodes in the MPP architecture database cluster according to a predefined sharding strategy (such as hash sharding, range sharding, etc.). MPP architecture databases (such as ClickHouse, GBase, etc.) possess a "sharenothing" characteristic, where each node independently manages its local data and computing resources, achieving collaborative computing through high-speed network interconnection. During the SQL parsing phase, the system first performs syntax analysis, semantic checking, and logical optimization on the user-submitted SQL statements, and then generates a distributed execution plan based on the data distribution. This execution plan decomposes the SQL operations into multiple subtasks and pushes these subtasks down to nodes containing relevant data, achieving parallel computing.

[0064] SQL parsing requires consideration of factors such as query complexity, data distribution granularity, and node load balancing. For example, the choice of sharding key should ensure even data distribution to avoid data skew. During execution plan generation, the system evaluates the CPU utilization, memory usage, and disk I / O throughput of each node to determine the optimal task allocation. Furthermore, MPP architecture databases typically support parallelism parameters (such as `max_threads` and `parallel_tasks`) to control the number of threads or tasks executed concurrently on each node, thereby optimizing overall computational efficiency.

[0065] This step is widely used in scenarios such as real-time risk analysis, premium calculation, and claims data aggregation in the insurance industry. For example, in daily scheduled batch processing tasks, the system can distribute incremental data accumulated in Kafka to the MPP cluster, generate parallel execution plans through SQL parsing, and achieve rapid processing of hundreds of millions of records. In real-time scenarios, such as when a customer's policy status changes, the system can immediately push the relevant data stream down to the MPP nodes for real-time statistical analysis, improving business response speed.

[0066] By intelligently generating SQL parsing and distributed execution plans, the parallel efficiency and resource utilization of data processing are significantly improved, effectively avoiding the performance bottlenecks of traditional centralized databases in large-scale data processing. Simultaneously, this mechanism supports dynamic expansion, adapting to the continuously growing business needs of the insurance industry and achieving high-throughput, low-latency data processing capabilities.

[0067] Furthermore, S3 includes: S31 reads JSON data streams from the message queue via Kafka Stream and converts them into a columnar storage format that can be processed by the MPP architecture.

[0068] Specifically, in this application proposal, the step of "reading JSON data streams from the message queue via Kafka stream and converting them into a columnar storage format that can be processed by the MPP architecture" is a key step in realizing integrated stream and batch data processing. The core of this step lies in performing structured parsing and format conversion on the real-time JSON data stream in Kafka to adapt to the columnar storage model of MPP architectures (such as ClickHouse, GBase, etc.), thereby improving the efficiency and accuracy of batch data processing.

[0069] As a stream processing engine, Kafka Stream first subscribes to a data stream for a specified topic from the Kafka message queue. Each message is encapsulated in JSON format, containing the fields and values ​​of the business entity. Kafka Stream parses each JSON message by defining a topology, extracting the field information and mapping it to a predefined columnar structure in the MPP database. This process typically includes schema parsing, field type conversion, and data cleaning. For example, if the original JSON contains the field `"policyNo": "1234567890"`, Kafka Stream will recognize it as a string type and ensure its compatibility with the `policyNoVARCHAR` field in the MPP table structure during the conversion process.

[0070] Configuring Kafka streams requires considering key metrics such as throughput, message processing latency, and partitioning strategies. For example, `application.id` identifies the streaming application instance, `bootstrap.servers` specifies the Kafka cluster address, and `key.serde` and `value.serde` define the key-value serialization method, typically `StringSerde` and `JsonSerde`. Furthermore, to ensure data consistency, `processing.guarantee` should be set to `exactly_once`, and `state.dir` should be configured for local state storage. During data transformation, field mapping rules also need to be defined, such as using Avro Schema or JSON Schema to standardize field types and order, ensuring consistency in the columnar storage format of data within MPP.

[0071] This step is widely used in real-time data aggregation and batch analysis scenarios in the insurance industry. For example, in business processes such as policy changes, claims events, and customer behavior, OLTP data is synchronized in real time via Kafka. Kafka Stream is responsible for converting this data into a columnar format that can be processed by MPP (Multi-Purpose Processing) for subsequent statistical analysis, report generation, or machine learning feature extraction. This process is typically deployed on a separate stream processing node, decoupled from the business database, to avoid impacting the performance of the transaction system.

[0072] This step enables efficient conversion from JSON format to columnar storage, providing structured and consistent data input for the MPP architecture, thereby significantly improving the execution efficiency and resource utilization of batch processing tasks. Through Kafka Streams' streaming capabilities, the system can continuously push data to MPP nodes with low latency and high throughput, providing a solid foundation for real-time analysis and decision support in the insurance industry.

[0073] S32, based on the SQL parsing results, splits the computation task into multiple subtasks, and pushes the subtasks down to the MPP node containing the corresponding data partition according to the data distribution characteristics.

[0074] Specifically, in this technical solution, the computation task is divided into multiple subtasks based on the SQL parsing results, and these subtasks are pushed down to MPP nodes containing the corresponding data partitions according to the data distribution characteristics. The core of this step lies in using SQL semantic analysis and data distribution metadata to transform the global query task into multiple local subtasks that can be executed in parallel, thereby fully leveraging the parallel computing capabilities of the MPP architecture.

[0075] In some implementations, this step first involves a SQL parser performing syntactic and semantic analysis on the user-submitted SQL statement to identify key information such as tables, fields, filtering conditions, and join operations. The parser is typically built on ANTLR or a similar syntax parsing framework, supporting SQL-92 and higher standard syntax. Subsequently, the system obtains the data partitioning information of each table on the MPP nodes through the distributed metadata management module, including the partition key, partitioning strategy (such as Range, Hash, List), and the data range handled by each node. For example, if a table uses Hash partitioning and its partition key is `policy_id`, the system can map query conditions to the corresponding partition nodes based on the value range of this field.

[0076] Furthermore, the system breaks down the original SQL into multiple subquery tasks based on the range of partition key values ​​involved in the SQL. Each subquery operates only on the dataset of a specific partition, thereby reducing unnecessary data scanning and network transmission. For example, if the SQL contains `WHERE policy_id IN (10001, 10002, 10003)`, and the hash partition distribution of `policy_id` is `10001 → Node1`, `10002 → Node2`, and `10003 → Node3`, then the system breaks down the SQL into three subtasks, which are pushed down to Node1, Node2, and Node3 for execution, respectively.

[0077] Key parameters involved in this step include SQL parsing depth, partition key matching accuracy, and task splitting granularity. SQL parsing depth determines whether complex subqueries and nested expressions can be identified; partition key matching accuracy affects the accuracy of task pushdown; and task splitting granularity determines the balance between parallelism and resource utilization. Generally, it is recommended to control the task splitting granularity between 1000 and 5000 records to avoid excessive scheduling overhead due to overly fine-grained tasks.

[0078] In practical applications, this step is widely used in batch data processing tasks in the insurance industry, such as policy statistics, premium calculation, and risk analysis. By pushing tasks down to the MPP node containing the corresponding data partition, the system can significantly reduce cross-node data transmission, improve query response speed, and reduce overall computing resource consumption.

[0079] The technical effect of this step is that, through the intelligent matching of SQL parsing and data distribution characteristics, it enables precise task pushdown and parallel execution, thereby improving the resource utilization and computing efficiency of the MPP architecture and solving the performance bottleneck problem of traditional centralized databases when processing large-scale data.

[0080] S4 monitors the status of database nodes through the federated cluster management component. When the primary node fails, it automatically switches to the backup node and feeds back the batch processing results to the front-end application.

[0081] Specifically, in this patented solution, the real-time monitoring of database node status through a federated cluster management component, and the automatic switchover to a backup node when the primary node fails, along with the feedback of batch processing results to the front-end application, are key steps in achieving high system availability and a closed-loop data processing system. This step in the distributed database architecture bears the dual responsibility of disaster recovery switching and result feedback.

[0082] Federated cluster management components (such as the CM component included with openGauss) monitor the status of the master node and slave nodes through a heartbeat mechanism. Specifically, CM periodically sends health check requests to the master node. If no response is received within a preset time (e.g., a timeout threshold is reached), the master node will be notified. If the time interval is less than 1 second, the primary node is considered to have failed. At this point, CM will trigger a failover process, automatically redirecting the current request to the backup node to ensure business continuity. In the JDBC connection string, by setting the `targetServerType=master` and `autoBalance=true` parameters, priority access to the primary node by the client and automatic load balancing in case of failure can be achieved.

[0083] Furthermore, this step also involves a feedback mechanism for batch processing results. After the MPP architecture (such as ClickHouse or GBase) completes the batch computation task, its execution engine sends the result set to the front-end application through a predefined interface (such as a RESTful API or a Kafka producer). During the feedback process, the system uses asynchronous communication to avoid blocking front-end business logic. The feedback data format is typically JSON or Parquet, supporting efficient transmission of structured and semi-structured data. In some implementations, the feedback latency is controlled within... Within seconds, to meet the insurance industry's requirements for data timeliness.

[0084] In application scenarios, this step is widely used between core transaction systems and batch analysis systems in the insurance industry. For example, in scenarios such as batch processing of insurance policies, aggregation of claims data, and training of risk models, when the primary database node becomes unavailable due to hardware failure or network interruption, the system can seamlessly switch to the backup node, while ensuring that batch processing results are promptly fed back to the front-end display or decision-making system, thereby guaranteeing business continuity and data availability.

[0085] This step yielded significant technical results, not only improving the system's disaster recovery and availability but also enhancing real-time data processing responsiveness through efficient batch processing and result feedback mechanisms. Combined with the parallel computing advantages of the MPP architecture, this step effectively resolved the performance bottleneck of traditional centralized databases in high-concurrency, high-data-volume scenarios, enabling the insurance business system to operate stably and process efficiently in a distributed environment.

[0086] Furthermore, S4 includes: S41 utilizes the CM cluster management component included with openGauss to monitor the CPU utilization, memory usage, and disk I / O throughput of each node in real time.

[0087] Specifically, in this application, the step of "using openGauss's built-in CM (Cluster Manager) cluster management component to monitor the CPU utilization, memory usage, and disk I / O throughput of each node in real time" is a key step in achieving high availability and load balancing of the distributed database. This step uses openGauss's CM (Cluster Manager) component to collect and analyze the resource status of each node in the distributed database cluster in real time, thereby providing data support for system scheduling, failover, and performance optimization.

[0088] The CM component, as the core cluster management module of openGauss, is responsible for centralized monitoring and management of the operational status of database instances. It periodically collects system resource metrics, including CPU utilization (expressed as a percentage, range...), through Agent processes deployed on each database node. ), memory usage ( The data includes disk I / O throughput (in MB / s or IOPS). The data collection frequency can be configured to once per second or once per minute, depending on the system's real-time requirements. The collected data is uploaded to the CM Server via the CM's communication protocol (such as TCP / IP or gRPC) for centralized analysis and visualization.

[0089] The CM component supports the configuration of various monitoring parameters, such as CPU utilization thresholds. Memory usage threshold and disk I / O over-limit alarm values When the CPU utilization of any node exceeds Or memory usage exceeds In such cases, CM can trigger an alarm mechanism and mark the node's status as "abnormal," thereby preventing it from continuing to undertake high-load tasks. Furthermore, CM supports scoring the health status of nodes, using the following formula:

[0090] in Indicates the first Health score of each node The weighting coefficients for each indicator are typically set to... This reflects the CPU's greater sensitivity to system performance.

[0091] This monitoring mechanism is widely used in distributed database environments within the insurance industry, particularly in high-concurrency transaction systems. Through real-time monitoring by CM, system administrators can promptly identify performance bottlenecks, such as nodes experiencing response delays due to excessive disk I / O throughput, enabling load balancing or failover operations. Furthermore, this monitoring data can be used for routing decisions in PDFC-Sharding; for example, when a node is overloaded, new requests can be automatically routed to nodes with lower loads, achieving dynamic load distribution.

[0092] This step significantly improves the stability and availability of the database cluster. Through real-time monitoring, the system can issue early warnings before node failures occur, preventing service interruptions. Simultaneously, combined with a health scoring mechanism, intelligent scheduling and resource optimization can be achieved, improving overall resource utilization and reducing operational costs. This technical solution demonstrates good adaptability and scalability in big data processing scenarios within the insurance industry, providing a solid foundation for building a highly available, high-performance distributed data processing architecture.

[0093] S42, when a primary node failure is detected, the optimal node is selected from the backup nodes for primary / backup switching according to the load balancing strategy configured by the autoBalance parameter.

[0094] Specifically, when a primary node failure is detected, the system selects the optimal node from the backup nodes for primary / backup failover based on the load balancing strategy configured by the `autoBalance` parameter. This is one of the key steps in the distributed high-availability cluster design of this solution. This step is implemented based on the collaborative mechanism of the openGauss database's cluster management component CM (Cluster Manager) and the self-developed middleware PDFC-Sharding.

[0095] openGauss provides automatic fault detection and failover capabilities for primary and standby nodes. The CM component continuously monitors the operational status of the primary node, including key metrics such as heartbeat signals, service response time, and SQL execution status. Once a primary node anomaly is detected (such as heartbeat timeout or service unreachability), CM triggers the failover process. PDFC-Sharding, as a database middleware, identifies the currently connected primary node through the `targetServerType=master` parameter in the JDBC connection string. When the primary node fails, PDFC-Sharding, based on the configured `autoBalance` parameter and load balancing strategies (such as weighted round-robin, minimum connection count, and response time priority), selects the optimal node from multiple standby nodes as the new primary node for connection switching. This process requires no manual intervention, and the switchover time is typically controlled within 30 seconds, ensuring business continuity.

[0096] The `autoBalance` parameter can be set to `true` or `false`. When set to `true`, PDFC-Sharding enables a load balancing mechanism, dynamically adjusting connection strategies based on the standby node's load status (such as CPU utilization, memory usage, and number of connections). The health status of the standby node is evaluated by the CM component through a heartbeat detection mechanism (default interval is 5 seconds). If three consecutive heartbeats fail, the primary node is considered to have failed. During failover, the system prioritizes nodes with lower load and shorter response times to minimize the impact on business operations.

[0097] This step is widely used in core transaction systems within the insurance industry, especially in scenarios with high concurrency and high availability requirements, such as policy processing, claims calculation, and customer information management. Through the master-slave failover mechanism, the system can quickly restore service when the master node fails or responds abnormally, avoiding business interruption and ensuring system stability.

[0098] The technical benefit of this step is a significant improvement in the system's disaster recovery capabilities and availability. By combining automatic detection with load balancing strategies, the system can respond quickly when a failure occurs, selecting the optimal backup node to take over the service, thereby reducing failure recovery time, improving resource utilization, and effectively avoiding resource contention issues.

[0099] Also includes: S5 performs data verification operations on the data stream transmitted to the MPP architecture. By comparing the differences between the primary key hash values ​​of the source database and the target MPP node, a compensation mechanism is used to retransmit lost or abnormal data.

[0100] Specifically, in some implementations, data verification is performed on the data stream transmitted to the MPP architecture by comparing the differences in primary key hash values ​​between the source database and the target MPP node to ensure the integrity and consistency of the data during transmission. This step is technically implemented through the collaborative work of the data synchronization tool exBase and the message queue Kafka, combining hash verification and compensation mechanisms to achieve automatic retransmission of abnormal or lost data.

[0101] The source database (such as openGauss) uses the exBase tool to capture data change events in real time, including insert, update, and delete operations. The captured data is encapsulated in JSON format and transmitted via a Kafka message queue. In the Kafka stream, the data stream is parsed and hashed according to the primary key field, generating a set of hash values. Meanwhile, after receiving the data, the target MPP node performs a hash calculation on the primary key field of each record to generate a local set of hash values. By comparison and Based on the differences, the system can identify records that were not successfully transmitted or have abnormal data.

[0102] The hash algorithm can use SHA-256 or MD5 to ensure the uniqueness and security of the hash value. The choice of the primary key field should be based on the uniqueness constraints of the business table, typically a business primary key or a composite primary key. The verification frequency can be set according to business needs, such as performing a hash comparison every 5 minutes, or triggering verification immediately after data stream processing is completed. The compensation mechanism can use a retransmission queue or rollback log to resend the difference records to the target MPP node for processing.

[0103] This step is applicable to core data synchronization scenarios in the insurance industry, such as the real-time synchronization and batch processing of critical business data like policy information, claims records, and customer data. In an MPP architecture, this data is used for analytical tasks such as generating reports, risk analysis, and customer profiling, requiring extremely high data integrity.

[0104] The technical advantage of this step lies in its ability to effectively address potential data loss, duplication, or inconsistency issues during transmission between heterogeneous systems through hash verification and compensation mechanisms, thereby improving the reliability and accuracy of data synchronization. Simultaneously, this mechanism enables real-time monitoring and repair of the data stream without significantly increasing system overhead, enhancing the overall architecture's robustness and fault tolerance.

[0105] The data processing method of this invention, by constructing a stream-batch integrated processing system that combines openGauss-based distributed database middleware with MPP architecture, effectively solves the performance bottleneck and resource contention problems of traditional centralized databases in high-concurrency, high-data-volume scenarios, and improves the real-time performance, scalability and high availability of data processing in the insurance industry.

[0106] Example 2 This invention, based on a domestically developed open-source database, forms a comprehensive distributed data processing solution, such as... Figure 2 As shown, based on the openGauss open-source database and distributed database middleware, massive amounts of business data and access pressure are distributed across different database nodes. Through real-time data synchronization, data changes are accurately captured without impacting database and business system performance, and analytical feature business data is aggregated into the MPP architecture. This addresses the scalability and performance bottlenecks of existing database products, while leveraging parallel data processing components to enhance high-concurrency, large-volume streaming and batch data computation capabilities, significantly improving real-time business analysis and computation capabilities.

[0107] 1) Distributed High-Availability Cluster Design: A distributed database system is implemented based on the open-source centralized database OpenGauss, incorporating the database sharding component PDFC-Sharding. OpenGauss-JDBC is integrated into PDFC-Sharding. The driver in pom.xml is replaced with OpenGauss-JDBC; a connection string instance is then obtained.

[0108] By modifying the database driver and connection string in PDFC-sharding, you can connect to the openGauss database. Adding `ShardingKey: ShardingValue` to the application system's request header, PDFC-sharding retrieves the `ShardingKey` from the request header and forwards the request to the corresponding sharded cluster using custom rules, such as a hash algorithm, to achieve vertical database sharding. In the openGauss JDBC connection string, the `targetServerType` parameter is set to "master" or "slave" to identify the master or slave node in the cluster. Simultaneously, openGauss comes with a built-in centralized management component (CM) that monitors the cluster's operational status. When the master node fails, it can automatically switch to a slave node, achieving high availability for the application. Figure 3 As shown.

[0109] 2) Streaming Data Processing Framework: In this solution, the streaming processing framework is implemented using tools such as the data synchronization tool exBase, the message queue Kafka, and the streaming tool KafkaStream. The openGauss distributed database cluster primarily handles OLTP-type business, while OLAP-type business data tables are configured in the data synchronization tool exBase. exBase monitors this data, captures real-time changes in the database, and synchronizes the data to the message queue Kafka in JSON format, thereby achieving heterogeneous data synchronization. KafkaStream reads data from Kafka, transforms this data into a data stream, and then transfers the data stream to the downstream MPP architecture, thus realizing a real-time streaming processing architecture, such as... Figure 4 As shown.

[0110] 3) Batch Business Processing: The batch business processing architecture in this solution mainly uses MPP architecture products, such as Clinkhouse and Gbase, to distribute data transmitted from Kafka streams evenly across each node in the MPP architecture. SQL parsing generates corresponding distributed execution plans, which are then pushed down to the nodes containing that data for computation and analysis. Finally, the results are fed back to the front-end application. The MPP architecture can process not only real-time analytical data but also scheduled tasks and T+1 tasks commonly used in the financial industry, such as payroll calculations. This framework mitigates the risk of competition for existing technology resources and, due to its strong scalability, solves the performance bottleneck caused by continuously growing data.

[0111] In summary, the distributed data processing method for the insurance industry presented in this invention is based on the open-source domestic database openGauss and incorporates the database middleware PDFC-Sharding to achieve a distributed data processing solution, addressing the issues of insufficient resources and poor scalability in centralized databases. Considering the high complexity and strong reliance on existing technologies in the financial and insurance industry, the method designs OLTP and OLAP business separation, processing these business processes within a suitable architecture. For OLTP-type businesses, a distributed relational database processing solution is used, while for OLAP-type businesses, an MPP architecture is employed. Furthermore, for businesses with high real-time requirements, data is transmitted to the MPP architecture for processing via a real-time streaming architecture, resolving issues of poor timeliness and heavy optimization tasks. Utilizing the built-in cluster management component of the openGauss database and employing database middleware, a federated cluster management architecture is formed, enabling high availability within the cluster. Real-time data is synchronized from the database to the heterogeneous MPP architecture, achieving real-time data streaming processing.

[0112] Example 3 This invention also provides a distributed data processing device 10 for the insurance industry, such as... Figure 5 As shown, the device 10 includes: The distributed database middleware building module is used to build distributed database middleware based on domestic centralized databases. It uses an intelligent routing mechanism to shard business data to multiple database nodes according to preset rules, thereby achieving vertical database sharding and load balancing. The real-time data capture and synchronization module is used to capture real-time changes in OLTP business data in a distributed database, convert the changed data into a standardized data format and transmit it to a message queue, forming a heterogeneous data synchronization channel. The data stream distribution and parallel computing module is used to distribute the data stream in the message queue to the MPP architecture database cluster, generate a distributed execution plan based on SQL parsing and push it down to the corresponding nodes for parallel computing; The federated cluster monitoring and failover module is used to monitor the status of database nodes through the federated cluster management component. When the primary node fails, it automatically switches to the backup node and feeds back the batch processing results to the front-end application.

[0113] Furthermore, the distributed database middleware building module is also used for: By adding the `ShardingKey: ShardingValue` identifier to the request header of the application system, the PDFC-Sharding middleware parses the identifier and uses a HASH algorithm to determine the target database sharding cluster. Configure the `targetServerType` parameter in the JDBC connection string to either master or slave, and combine it with the `autoBalance` parameter to achieve automatic load balancing and failover between the master and slave nodes.

[0114] Furthermore, the real-time data capture and synchronization module is also used for: The exBase tool is used to capture the underlying logs of the openGauss database in real time, and incremental data change records are generated by parsing the DML operations in the binlog logs. The incremental data is serialized using JSON format, and asynchronous transmission of heterogeneous data across systems is achieved through a Kafka message queue.

[0115] Furthermore, the data stream distribution and parallel computing module is also used for: Read JSON data streams from the message queue using Kafka Stream and convert them into a columnar storage format that can be processed by the MPP architecture; The computation task is divided into multiple subtasks based on the SQL parsing results, and the subtasks are pushed down to the MPP node containing the corresponding data partitions according to the data distribution characteristics.

[0116] This invention discloses a distributed data processing device for the insurance industry. By comparing and verifying the primary key hash value of the data stream transmitted to the MPP architecture and introducing a compensation mechanism to achieve automatic retransmission of abnormal data, the device further enhances the integrity and consistency of data transmission, effectively reduces the risk of data loss, and improves the data reliability and system fault tolerance of the insurance industry in integrated batch processing.

[0117] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0118] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0119] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A distributed data processing method for the insurance industry, characterized in that, include: S1 builds a distributed database middleware based on a domestic centralized database. Through an intelligent routing mechanism, business data is sharded to multiple database nodes according to preset rules to achieve vertical database sharding and load balancing. S2, capture real-time changes of OLTP business data in the distributed database, convert the changed data into a standardized data format and transmit it to the message queue to form a heterogeneous data synchronization channel; S3, distribute the data stream in the message queue to the MPP architecture database cluster, generate a distributed execution plan based on SQL parsing and push it down to the corresponding nodes for parallel computing; S4 monitors the status of database nodes through the federated cluster management component. When the primary node fails, it automatically switches to the backup node and feeds back the batch processing results to the front-end application.

2. The method as described in claim 1, characterized in that, The construction of a distributed database middleware based on a domestically developed centralized database, which uses an intelligent routing mechanism to shard business data to multiple database nodes according to preset rules to achieve vertical database sharding and load balancing, also includes: S11, by adding the ShardingKey: ShardingValue identifier to the request header of the application system, the PDFC-Sharding middleware parses the identifier and uses the HASH algorithm to determine the target database sharding cluster; S12, configure the targetServerType parameter in the JDBC connection string to master or slave, and combine it with the autoBalance parameter to realize automatic load balancing and failover of master and slave nodes.

3. The method as described in claim 1, characterized in that, The process of capturing real-time changes to OLTP business data in the distributed database, converting the changed data into a standardized data format, and transmitting it to a message queue to form a heterogeneous data synchronization channel also includes: S21, use the exBase tool to capture the underlying logs of the openGauss database in real time, and generate incremental data change records by parsing the DML operations in the binlog logs; S22, the incremental data is serialized using JSON format, and asynchronous transmission of heterogeneous data across systems is achieved through Kafka message queue.

4. The method as described in claim 1, characterized in that, The step of distributing the data stream in the message queue to the MPP architecture database cluster, generating a distributed execution plan based on SQL parsing, and pushing it down to the corresponding nodes for parallel computing also includes: S31 reads JSON data streams from the message queue via Kafka stream and converts them into a columnar storage format that can be processed by the MPP architecture. S32, based on the SQL parsing results, splits the computation task into multiple subtasks, and pushes the subtasks down to the MPP node containing the corresponding data partition according to the data distribution characteristics.

5. The method as described in claim 1, characterized in that, The method of monitoring the database node status through the federated cluster management component, automatically switching to the backup node when the primary node fails, and simultaneously feeding back the batch processing results to the front-end application also includes: S41 utilizes the CM cluster management component built into openGauss to monitor the CPU utilization, memory usage, and disk I / O throughput of each node in real time. S42, when a primary node failure is detected, the optimal node is selected from the backup nodes for primary / backup switching according to the load balancing strategy configured by the autoBalance parameter.

6. The method as described in claim 1, characterized in that, Also includes: S5 performs data verification operations on the data stream transmitted to the MPP architecture. By comparing the differences between the primary key hash values ​​of the source database and the target MPP node, a compensation mechanism is used to retransmit lost or abnormal data.

7. A distributed data processing device for the insurance industry, characterized in that, include: The distributed database middleware building module is used to build distributed database middleware based on domestic centralized databases. It uses an intelligent routing mechanism to shard business data to multiple database nodes according to preset rules, thereby achieving vertical database sharding and load balancing. The real-time data capture and synchronization module is used to capture real-time changes in OLTP business data in a distributed database, convert the changed data into a standardized data format and transmit it to a message queue, forming a heterogeneous data synchronization channel. The data stream distribution and parallel computing module is used to distribute the data stream in the message queue to the MPP architecture database cluster, generate a distributed execution plan based on SQL parsing and push it down to the corresponding nodes for parallel computing; The federated cluster monitoring and failover module is used to monitor the status of database nodes through the federated cluster management component. When the primary node fails, it automatically switches to the backup node and feeds back the batch processing results to the front-end application.

8. The apparatus as claimed in claim 7, characterized in that, The distributed database middleware building module is also used for: By adding the `ShardingKey: ShardingValue` identifier to the request header of the application system, the PDFC-Sharding middleware parses the identifier and uses a HASH algorithm to determine the target database sharding cluster. Configure the `targetServerType` parameter in the JDBC connection string to either master or slave, and combine it with the `autoBalance` parameter to achieve automatic load balancing and failover between the master and slave nodes.

9. The apparatus as claimed in claim 7, characterized in that, The real-time data capture and synchronization module is also used for: The exBase tool is used to capture the underlying logs of the openGauss database in real time, and incremental data change records are generated by parsing the DML operations in the binlog logs. The incremental data is serialized using JSON format, and asynchronous transmission of heterogeneous data across systems is achieved through a Kafka message queue.

10. The apparatus as claimed in claim 7, characterized in that, The data stream distribution and parallel computing module is also used for: Read JSON data streams from the message queue using Kafka Stream and convert them into a columnar storage format that can be processed by the MPP architecture; The computation task is divided into multiple subtasks based on the SQL parsing results, and the subtasks are pushed down to the MPP node containing the corresponding data partitions according to the data distribution characteristics.

Images