Intelligent flexible batch processing method for mass transaction of insurance receipts and payments and related equipment

Abstract

This invention relates to the field of insurance transaction data processing technology, specifically disclosing an intelligent elastic batch processing method and related equipment for massive insurance payment and receipt transactions. This invention employs JDBC batch processing to write multi-source insurance payment and receipt transaction data into a temporary table and assign a globally unique batch number. Based on a pre-configured verification rule table, batch verification SQL statements are dynamically generated and executed at the database layer for one-time verification. A distributed task scheduler dynamically segments the processing based on system resource load, and an asynchronous communication mechanism pushes task progress in real time. For failed tasks, retries and dead-letter queue management are automatically triggered. This solves the technical problems of traditional scheduled batch jobs, such as slow front-end rendering and heavy network transmission load caused by full page queries, excessive I / O overhead due to repeated database queries caused by multiple selections, and long waiting times for users caused by synchronous processing mechanisms. This improves the timeliness of batch processing for massive insurance payment and receipt transactions and optimizes the user experience.

Description

Technical Field

[0001] This invention relates to the field of insurance transaction data processing technology, and in particular to an intelligent and flexible batch processing method and related equipment for massive insurance payment and collection transactions. Background Technology

[0002] With the continuous expansion of the insurance business, the volume of payment and receipt transaction data is growing exponentially, making the traditional scheduled batch processing mode insufficient to meet the demand for efficient processing. Existing insurance industry payment and receipt transaction batch processing systems mainly adopt a scheduled batch processing mode based on commonly used SQL queries. The specific implementation process includes batch data querying, iterative verification, and multiple queries. This mode uses conventional SQL queries for data retrieval when processing large volumes of business data through the insurance business system, and retrieves detailed business information from the database during the business data verification stage. Due to the massive data volume and the need to perform business verification after each query, the system frequently needs to establish network connections with the database, resulting in a significant increase in network latency. Furthermore, the lengthy business process, requiring multiple iterative data queries throughout the process, further exacerbates network consumption and reduces overall processing efficiency.

[0003] Existing technologies objectively suffer from the following core defects: First, data loading bottlenecks. The page queries all data at once and hides the primary key, leading to slow front-end rendering and heavy network load, especially when the data volume reaches hundreds of thousands of records, resulting in extremely slow page response. Second, duplicate query issues. Each time data is selected, the backend re-queries the database based on the primary key, causing multiple I / O and computational overheads. Third, synchronous processing mechanisms. All operations are executed synchronously, requiring users to wait for extended periods, resulting in a poor user experience. These defects severely restrict the efficiency of batch processing for insurance payment transactions and the user experience, urgently requiring a new technical solution that can optimize data loading strategies, reduce duplicate queries, and achieve asynchronous elastic processing. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an intelligent, flexible batch processing method and related equipment for handling massive insurance payment and collection transactions.

[0005] Firstly, this invention provides an intelligent, flexible batch processing method for massive insurance payment and collection transactions. The technical solution of this method is as follows: In response to the request to import batch transaction data, the system reads the insurance payment and receipt transaction data to be processed from the multi-source data interface, assigns a globally unique batch number to the insurance payment and receipt transaction data, and uses JDBC batch processing to write the insurance payment and receipt transaction data corresponding to the batch number into a temporary table in batches. After the data is written, based on the pre-configured verification rule table, batch verification SQL statements are dynamically generated according to the batch number. At the database layer, a one-time batch verification is performed on the insurance payment and receipt transaction data with the batch number in the temporary table, and the verification results are written back to the temporary table in batches. After the verification is passed, a batch processing task is generated according to the batch number and submitted to the distributed task scheduler. The distributed task scheduler dynamically allocates execution nodes to perform fragmented processing of the insurance payment and receipt transaction data corresponding to the batch number based on the system resource load and task queue status. During the execution of the segmented processing task, the execution status corresponding to the batch number in the task status table is updated in real time, and task progress information is pushed to the front end through an asynchronous communication mechanism; When the fragmentation processing task is detected to have failed, a retry mechanism is automatically triggered according to the failure type. After a preset number of retries, the fragmentation processing task is transferred to a dead letter queue and the exception information is recorded.

[0006] The beneficial effects of the intelligent and flexible batch processing method for massive insurance payment and collection transactions of the present invention are as follows: The method of this invention uses JDBC batch processing to write multi-source insurance payment and receipt transaction data into a temporary table and assign a globally unique batch number. Based on a pre-configured verification rule table, batch verification SQL statements are dynamically generated and executed at the database layer for one-time verification. A distributed task scheduler dynamically segments the processing according to system resource load, pushes task progress in real time with an asynchronous communication mechanism, and automatically triggers retries and dead-letter queue management for failed tasks. This solves the technical problems of traditional scheduled batch jobs, such as full page queries causing slow front-end rendering and heavy network transmission load, repeated database queries caused by multiple selections resulting in excessive I / O overhead, and synchronous processing mechanisms forcing users to wait for a long time. It improves the timeliness of batch processing of massive insurance payment and receipt transactions and optimizes the interactive experience.

[0007] Based on the above solution, the intelligent elastic batch processing method for massive insurance payment and collection transactions of the present invention can be further improved as follows.

[0008] In one alternative approach, the step of reading insurance payment and collection transaction data to be processed from a multi-source data interface in response to a bulk transaction data import request includes: The system receives externally initiated import requests through a predefined data import interface. The import requests carry the data source identifier of the batch transaction data to be processed. The corresponding data reading channel is determined from the multi-source data interface based on the data source identifier. The multi-source data interface includes a file data source interface, a message queue data source interface, and a business database table data source interface. The original transaction data is read from the corresponding data source through the data reading channel, and the original transaction data is parsed into the insurance payment and receipt transaction data.

[0009] The advantages of adopting the above optional method are as follows: It can further receive import requests carrying data source identifiers through a predefined data import interface, determine the data reading channel corresponding to the file data source, message queue data source or business database table data source based on the data source identifier, parse the original transaction data into insurance payment and receipt transaction data, realize standardized access and flexible adaptation of multi-source heterogeneous data, and enhance the channel expansion capability of data reading.

[0010] In one alternative approach, the steps of assigning a globally unique batch number to the insurance payment and receipt transaction data and using JDBC batch processing to write the insurance payment and receipt transaction data corresponding to the batch number into a temporary table include: A batch number generator is used to generate a globally unique batch number for the insurance payment and receipt transaction data, and a correspondence is established between the batch number and the insurance payment and receipt transaction data. The insurance payment and receipt transaction data with the same batch number are divided into multiple data blocks according to the preset batch size; The insurance payment and collection transaction data in each data block is added to the batch processing buffer using the JDBC addBatch method, and the data in the batch processing buffer is written to a temporary table all at once using the executeBatch method when the preset batch processing size is reached. During the writing process, the writing status of each data block is recorded, and in the event of a writing error, the incomplete data blocks are resumed from where they left off based on the writing status and the batch number.

[0011] The advantages of adopting the above optional method are as follows: further establishing the correspondence between batch number and transaction data through batch number generator, dividing data blocks according to preset batch processing size, using JDBC's addBatch and executeBatch methods to realize batch writing, and recording the writing status to support breakpoint resume, realizing high throughput of massive data into the database and breakpoint recovery in abnormal scenarios, and enhancing the stability and continuity of data writing.

[0012] In one alternative approach, after data writing is complete, based on a pre-configured verification rule table, batch verification SQL statements are dynamically generated according to the batch number. A one-time batch verification is then performed on the insurance payment and receipt transaction data with the batch number in the temporary table at the database layer, and the verification results are written back to the temporary table in batches. This includes the following steps: After all the insurance payment and receipt transaction data is written to the temporary table, the set of verification rules corresponding to the insurance payment and receipt transaction data is read from the verification rule table. The verification rule table includes field names, verification types, and verification SQL templates. Based on the batch number, the field name, the verification type, and the verification SQL template, the batch verification SQL statement is dynamically generated. The batch verification SQL statement is used to perform legality verification on all insurance payment and receipt transaction data of the batch number in a single database scan based on the field name and the verification type. The batch verification SQL statement is sent to the database layer for execution. The database layer performs a one-time batch scan on the insurance payment and receipt transaction data with the batch number in the temporary table and returns a set of verification results. Based on the verification result set, generate the error code and error description corresponding to each insurance payment and receipt transaction data, and write the error code and error description back to the verification result field corresponding to each insurance payment and receipt transaction data in the temporary table in batches as verification results.

[0013] The advantages of adopting the above optional method are as follows: further reading the field name, verification type and verification SQL template from the verification rule table, dynamically concatenating and generating batch verification SQL statements and performing a one-time batch scan at the database layer, and writing back the error code and error description in batch to the verification result field of the temporary table, realizing the dynamic configuration of the verification logic and one-time batch verification at the database layer, reducing the multiple network round-trip overhead caused by verification one by one.

[0014] In one optional approach, after successful verification, a batch processing task is generated based on the batch number and submitted to a distributed task scheduler. The distributed task scheduler then dynamically allocates execution nodes based on system resource load and task queue status to perform fragmented processing of the insurance payment and receipt transaction data corresponding to the batch number. This includes: After the verification results are written back to the temporary table in batches, it is determined whether all insurance payment and receipt transaction data with the batch number have passed the legality verification. If all validations pass, a batch processing task is created based on the batch number, and the batch processing task is submitted to the distributed task scheduler. The system resource load and task queue status are obtained through the distributed task scheduler. The system resource load includes CPU utilization and memory utilization, and the task queue status includes the number of tasks to be processed and the queuing time. Based on the system resource load and the task queue status, a target execution node is dynamically selected from multiple candidate execution nodes, and the batch processing task is assigned to the target execution node; Through the target execution node, the insurance payment and receipt transaction data corresponding to the batch number is divided into multiple sub-tasks according to the preset sharding dimensions. Each sub-task corresponds to a data shard. The sharding dimensions include at least one of the following: institution dimension, product dimension, and date dimension. The multiple subtasks are distributed to multiple worker threads corresponding to the target execution node for parallel execution.

[0015] The advantages of adopting the above optional method are as follows: After the verification is passed, a batch processing task is created and submitted to the distributed task scheduler. The target execution node is dynamically selected based on CPU utilization, memory utilization and task queue status. Subtasks are divided according to the organizational dimension, product dimension or date dimension and distributed to multiple worker threads for parallel execution. This realizes dynamic scheduling based on resource load and parallel processing of multi-dimensional data sharding, and improves the balance and concurrency efficiency of task execution.

[0016] In one alternative approach, the steps of updating the execution status corresponding to the batch number in the task status table in real time during the execution of the segmented processing task, and pushing task progress information to the front end through an asynchronous communication mechanism, include: When the multiple subtasks are executed in parallel, a subtask status record is created for each subtask, and the execution status changes of each subtask are captured in real time through an event listening mechanism. Based on the changes in the execution status of all subtasks, the overall execution status corresponding to the batch number is aggregated and generated; The overall execution status is written in real time to the execution status field corresponding to the batch number in the task status table, and the task progress percentage is calculated based on the number of completed sub-tasks and the total number of sub-tasks. The task progress percentage and the overall execution status are pushed to the front end for display as task progress information through a pre-established asynchronous communication connection. When any subtask fails, the overall execution status is updated to failure, and a failure notification is pushed to the front end via the asynchronous communication connection.

[0017] The advantages of adopting the above optional approach are as follows: when subtasks are executed in parallel, state changes are captured through an event listening mechanism, the overall execution state is aggregated and written to the task status table in real time, the task progress percentage is calculated and pushed to the front end through an asynchronous communication connection, and the status is updated and a failure notification is pushed when a failure occurs. This realizes real-time aggregation of execution status and asynchronous push of task progress, improving the user's perception of the batch processing process and the interactive experience.

[0018] In one optional approach, when the fragmentation processing task is detected to have failed, a retry mechanism is automatically triggered based on the failure type, and after a preset number of retries, the fragmentation processing task is transferred to a dead-letter queue and the exception information is recorded. This includes: When any subtask fails to execute, the failure type corresponding to the subtask is obtained. The failure type is either a network timeout type, a data conflict type, or a system exception type. Based on the failure type, query the corresponding retry count threshold and retry interval from the pre-configured retry policy table; Within the retry threshold range, the subtask is automatically retried according to the retry interval, and the retry count is updated after each retry. When the retry count reaches the retry threshold and the subtask still fails to execute successfully, the data fragment corresponding to the subtask is transferred to the dead letter queue as a whole. Read the data fragments and failure types corresponding to the subtask from the dead-letter queue, generate the exception information containing the batch number, the subtask identifier and the reason for failure, and write the exception information into the exception record table.

[0019] The advantages of adopting the above optional method are as follows: when a subtask fails, the failure type is obtained, such as network timeout type, data conflict type, or system anomaly type. The retry policy table is queried to execute automatic retry and the task is transferred to the dead letter queue after reaching the threshold. An anomaly information containing batch number, subtask identifier, and failure reason is generated and written to the anomaly record table. This realizes the centralized management of differentiated retry policies and failed tasks, and improves the pertinence and traceability of anomaly handling.

[0020] Secondly, this invention provides an intelligent, flexible batch processing system for massive insurance payment and collection transactions. The technical solution of this system is as follows: The write module is used to respond to the import request of batch transaction data, read the insurance payment and receipt transaction data to be processed from the multi-source data interface, assign a globally unique batch number to the insurance payment and receipt transaction data, and use JDBC batch processing to write the insurance payment and receipt transaction data corresponding to the batch number into a temporary table in batches. The verification module is used to dynamically generate batch verification SQL statements based on the batch number according to the pre-configured verification rule table after the data is written, perform a one-time batch verification on the insurance payment and receipt transaction data with the batch number in the temporary table at the database layer, and write the verification results back to the temporary table in batches. The allocation module is used to generate batch processing tasks based on the batch number after the verification is passed and submit them to the distributed task scheduler. The distributed task scheduler dynamically allocates execution nodes to perform fragmented processing of the insurance payment and receipt transaction data corresponding to the batch number based on the system resource load and task queue status. The update module is used to update the execution status corresponding to the batch number in the task status table in real time during the execution of the segmented processing task, and push task progress information to the front end through an asynchronous communication mechanism. The detection module is used to automatically trigger a retry mechanism based on the failure type when the fragmentation processing task is detected to have failed, and to transfer the fragmentation processing task to a dead letter queue and record the abnormal information after the retry reaches a preset number of times.

[0021] The beneficial effects of the intelligent elastic batch processing system for massive insurance payment and collection transactions of the present invention are as follows: The system of this invention uses JDBC batch processing to write multi-source insurance payment and receipt transaction data into a temporary table and assign a globally unique batch number. Based on a pre-configured verification rule table, it dynamically generates batch verification SQL statements and performs one-time verification at the database layer. Through a distributed task scheduler, it dynamically segments the processing according to the system resource load, pushes task progress in real time with an asynchronous communication mechanism, and automatically triggers retries and manages dead-letter queues for failed tasks. This solves the technical problems of traditional scheduled batch jobs, such as full page queries causing slow front-end rendering and heavy network transmission load, repeated database queries caused by multiple selections resulting in excessive I / O overhead, and synchronous processing mechanisms forcing users to wait for a long time. It improves the timeliness of batch processing of massive insurance payment and receipt transactions and optimizes the interactive experience.

[0022] Thirdly, the technical solution of an electronic device according to the present invention is as follows: It includes a memory, a processor, and a program stored in the memory and running on the processor, wherein the processor executes the program to implement the steps of the intelligent elastic batch processing method for massive insurance payment transactions as described in this invention.

[0023] Fourthly, the technical solution of a computer-readable storage medium provided by the present invention is as follows: The computer-readable storage medium stores instructions that, when read, cause the computer-readable storage medium to perform the steps of the intelligent elastic batch processing method for massive insurance payment transactions of the present invention.

[0024] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0025] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a flowchart illustrating an embodiment of an intelligent, flexible batch processing method for massive insurance payment transactions according to the present invention. Figure 2 This is a schematic diagram of an embodiment of an intelligent elastic batch processing system for massive insurance payment and collection transactions according to the present invention. Figure 3 This is a schematic diagram of an embodiment of an electronic device according to the present invention. Detailed Implementation

[0026] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0027] Figure 1 This diagram illustrates a flowchart of an embodiment of an intelligent, flexible batch processing method for massive insurance payment and collection transactions provided by the present invention. This method can be executed by electronic devices such as terminal devices or servers. The terminal device can be any fixed or mobile terminal, such as user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, or wearable device. The server can be a single server or a server cluster consisting of multiple servers. Any electronic device can implement the intelligent, flexible batch processing method for massive insurance payment and collection transactions by having its processor call computer-readable instructions stored in its memory. Figure 1 As shown, it includes the following steps: S1. In response to the request to import batch transaction data, read the insurance payment and receipt transaction data to be processed from the multi-source data interface, assign a globally unique batch number to the insurance payment and receipt transaction data, and use JDBC batch processing to write the insurance payment and receipt transaction data corresponding to the batch number into a temporary table in batches.

[0028] Import requests refer to instructions initiated by external systems to import batch transaction data into the batch processing system. These instructions include basic information such as the data source. For example, on January 1, 2026, Insurance Company A's payment system initiates an import request to import the pending auto insurance premium payment data for that day into the batch processing system. Multi-source data interfaces refer to unified access points provided by the batch processing system for reading data from different data sources, supporting various data source types such as files, message queues, and database tables. For example, the payment system uses a multi-source data interface to simultaneously read pending premium transaction data from an auto insurance renewal Excel file, a real-time payment message queue, and a historical arrears table in the business database. Insurance payment transaction data refers to transaction records generated by insurance companies in their payment business related to premium collection and claims payment. For example, the 10,000 auto insurance premium payment transaction records imported this time each record includes information such as policy number, policyholder, amount of premium receivable, and payment period.

[0029] The batch number refers to a globally unique identifier assigned to a batch of imported insurance payment and collection transaction data. This identifier is used to track and manage the data throughout the entire batch processing lifecycle. For example, batch number B20260101001 is assigned to the auto insurance premium data imported on January 1, 2026. JDBC batch processing refers to a method that uses Java Database Connectivity to send multiple data records to the database for execution at once, reducing the number of network interactions. For example, using JDBC batch processing, 10,000 auto insurance premium records with the same batch number B20260101001 can be merged into batches of 2,000 records each, and written to a temporary table using the `addBatch` and `executeBatch` methods. A temporary table is a table in the database used to temporarily store batch data being processed; it can be cleared or deleted after processing is complete. For example, the auto insurance premium data corresponding to batch number B20260101001 can be written to a temporary table named `temp_payment`, awaiting subsequent verification and processing.

[0030] S2. After the data is written, based on the pre-configured verification rule table, batch verification SQL statements are dynamically generated according to the batch number. In the database layer, a one-time batch verification is performed on the insurance payment and receipt transaction data with the batch number in the temporary table, and the verification results are written back to the temporary table in batches.

[0031] The validation rule table refers to a predefined table in the database used to store metadata for various data validation rules, including the fields to which the rules apply, the validation type, and the corresponding SQL template. For example, the validation rule table stores rules for car insurance premium data: the "policy number" field has a validation type of NOT NULL, and the "premium amount" field has a validation type of greater than 0, each associated with a corresponding validation SQL template. Batch validation SQL statements refer to one or more SQL statements dynamically generated from the validation rule table, capable of performing multiple validity checks on the entire batch of data in a single database scan. For example, based on batch number B20260101001 and the validation rule table, the following SQL statement can be dynamically generated: SELECT batch_id, COUNT(*) FROM temp_payment WHERE batch_id='B20260101001' AND (policy_no IS NULL OR premium_amount<=0) GROUP BY batch_id.

[0032] The database layer refers to the layer where the database management system resides, responsible for data storage, retrieval, and computation. In this solution, validation logic is devolved to the database layer to reduce data round trips between the application and the database. For example, batch validation SQL statements are sent to the database layer, where the database engine directly scans and validates the data in the temporary table, instead of retrieving and validating data record by record at the application layer. One-time batch validation means completing the validity check of all records in a batch of data with a single database operation, rather than performing multiple checks record by record. For example, the database layer performs a one-time batch validation on all car insurance premium data for batch number B20260101001, completing the non-null and amount validity checks of all records with a single scan. The validation result refers to whether each piece of data generated after batch validation is valid and the specific reason for any invalidity. For example, the result set returned by one-time batch validation includes batch number B20260101001 and the number of erroneous records detected, and further generates the error code and error description for each erroneous record.

[0033] S3. After the verification is passed, a batch processing task is generated according to the batch number and submitted to the distributed task scheduler. The distributed task scheduler dynamically allocates execution nodes to perform fragmented processing of the insurance payment and receipt transaction data corresponding to the batch number based on the system resource load and task queue status.

[0034] Batch processing tasks refer to job units that perform subsequent processing (such as sharding and execution) on a batch of insurance payment and receipt transaction data. They typically include a batch number and processing logic. For example, after successful verification, a batch processing task is created based on batch number B20260101001 to prepare for sharding this batch of auto insurance premium data. A distributed task scheduler is a distributed component used to manage and schedule batch processing tasks, dynamically allocating tasks to different nodes for execution based on resource availability. For example, using Elastic-Job as a distributed task scheduler, it receives the batch processing task corresponding to batch number B20260101001 and determines which execution node to assign the task to based on the current load of each node.

[0035] System resource load refers to the computing resource usage of execution nodes, typically including CPU and memory utilization. For example, before allocating task batch number B20260101001, the distributed task scheduler checks the CPU and memory utilization of each candidate execution node to avoid assigning the task to a node with excessive load. Task queue status refers to the number of tasks waiting to be processed in the distributed task scheduler and their queuing time. For example, if the scheduler sees 15 tasks waiting in the queue with an estimated maximum queuing time of 30 seconds, it dynamically determines the allocation strategy for new tasks based on the system resource load.

[0036] In this context, an execution node refers to a server instance in a distributed system that actually executes batch processing tasks. For example, the distributed task scheduler selects node B, which has the lowest current CPU utilization, from three candidate execution nodes A, B, and C as the target execution node to process the auto insurance premium data for batch number B20260101001. Sharding processing refers to splitting a batch processing task into multiple smaller subtasks according to a certain dimension and assigning them to multiple worker threads for parallel execution. For example, the target execution node splits the 10,000 auto insurance premium data entries corresponding to batch number B20260101001 into three data shards—Institution X, Institution Y, and Institution Z—according to the institutional dimension, each shard is processed in parallel by three worker threads.

[0037] S4. During the execution of the segmented processing task, update the execution status corresponding to the batch number in the task status table in real time, and push task progress information to the front end through an asynchronous communication mechanism.

[0038] The task status table refers to a table in the database used to record the current execution status of each batch processing task corresponding to each batch number, including statuses such as pending, verification, execution, completed, and failed. For example, a record is created in the task status table for batch number B20260101001, with the initial status being pending, and updated sequentially to verification, execution, and completed as processing progresses. Execution status refers to the stage a batch processing task or subtask is at at a given moment; for example, the current execution status of the batch processing task with batch number B20260101001 is execution, and one of its subtasks is completed.

[0039] Asynchronous communication mechanism refers to the non-blocking exchange of information between the system and the front-end, where the front-end can receive progress updates without waiting for the back-end to complete processing. For example, an asynchronous communication connection can be established with the front-end via WebSocket to push the processing progress of batch number B20260101001 to the operator interface in real time. The front-end refers to the user interface, typically a browser or client application, used to display task progress and receive user instructions. For example, the operator can see the real-time processing progress bar and current status of batch number B20260101001 in the browser front-end. Task progress information refers to the percentage of batch processing task completion and the current overall execution status. For example, pushing task progress information to the front-end: Batch number B20260101001 is currently 30% complete, and the overall execution status is "in progress".

[0040] S5. When the fragmentation processing task is detected to have failed, a retry mechanism is automatically triggered according to the failure type, and after a preset number of retries, the fragmentation processing task is transferred to a dead letter queue and the abnormal information is recorded.

[0041] The retry mechanism refers to the strategy whereby the system automatically retryes a task when it fails. For example, if a subtask in batch number B20260101001 fails due to a network timeout, the retry mechanism is automatically triggered to attempt to re-execute the subtask. The preset number of retries refers to the maximum number of retries pre-set in the retry mechanism; exceeding this number will result in the task being considered a final failure. For example, if the preset number of retries is configured to be 3, then each failed task can be retried a maximum of 3 times.

[0042] The dead-letter queue is a message queue used to store failed tasks that cannot be executed successfully after a preset number of retries, allowing for subsequent manual intervention or analysis. For example, if a subtask of batch number B20260101001 fails after three retries, the data fragment corresponding to that subtask is transferred to the dead-letter queue as a whole. Exception information refers to structured information recording relevant data and reasons for failure of failed tasks, facilitating problem tracing. For example, data from failed subtasks can be read from the dead-letter queue, generating exception information containing batch number B20260101001, subtask identifier, and reason for failure, and then written to an exception record table.

[0043] The technical solution of this embodiment uses JDBC batch processing to write multi-source insurance payment and receipt transaction data into a temporary table and assign a globally unique batch number. Based on a pre-configured verification rule table, batch verification SQL statements are dynamically generated and executed at the database layer for one-time verification. A distributed task scheduler dynamically segments the processing according to system resource load, pushes task progress in real time with an asynchronous communication mechanism, and automatically triggers retries and dead-letter queue management for failed tasks. This solves the technical problems of traditional scheduled batch jobs, such as full page queries causing slow front-end rendering and heavy network transmission load, repeated database queries caused by multiple selections resulting in excessive I / O overhead, and synchronous processing mechanisms forcing users to wait for a long time. It improves the timeliness of batch processing of massive insurance payment and receipt transactions and optimizes the interactive experience.

[0044] In one alternative approach, the step of reading insurance payment and collection transaction data to be processed from a multi-source data interface in response to a bulk transaction data import request includes: The system receives externally initiated import requests through a predefined data import interface. The import requests carry the data source identifier of the batch transaction data to be processed.

[0045] The data import interface refers to the programming interface provided by the system for receiving batch data import requests, which specifies the format and parameters of the requests. For example, Insurance Company A's payment system sends an import request containing a data source identifier to the batch processing system by calling the predefined data import interface. The data source identifier is a marker used to distinguish different data sources, such as a file path, message queue topic, or database table name. For example, the data source identifiers carried in the import request might be "excel:20260101_car_renewal.xlsx", "mq:premium_payment", and "db:history_arrears".

[0046] The corresponding data reading channel is determined from the multi-source data interface based on the data source identifier. The multi-source data interface includes a file data source interface, a message queue data source interface, and a business database table data source interface.

[0047] Among them, a data reading channel refers to a data reading connection or pipeline established for a specific data source, used to obtain raw data from that data source; for example, the corresponding data reading channel is determined according to the data source identifier: a file stream reading channel is created for an Excel file, a consumer channel is created for a message queue, and a JDBC query channel is created for a business database table.

[0048] The file data source interface refers to an interface specifically designed for reading file-type data sources, supporting formats such as Excel and CSV; for example, reading raw transaction data from a car insurance renewal Excel file using the file data source interface. The message queue data source interface refers to an interface specifically designed for consuming data from message queues, supporting formats such as Kafka and RabbitMQ; for example, pulling the most recent minute's premium payment data from a real-time payment message queue using the message queue data source interface. The business database table data source interface refers to an interface specifically designed for querying table data from a business database, typically requiring the specification of the table name and query conditions; for example, querying all outstanding car insurance premium records from the historical arrears table using the business database table data source interface.

[0049] The original transaction data is read from the corresponding data source through the data reading channel, and the original transaction data is parsed into the insurance payment and receipt transaction data.

[0050] Raw transaction data refers to the unparsed initial data format read from various data sources, which may be file lines, message bytes, or database row records; for example, each row of raw data read from an Excel file contains raw strings of fields such as policy number, policyholder, and premiums receivable.

[0051] In the above optional methods, an import request carrying a data source identifier is further received through a predefined data import interface. Based on the data source identifier, the data reading channel corresponding to the file data source, message queue data source, or business database table data source is determined. The original transaction data is parsed into insurance payment and receipt transaction data, realizing standardized access and flexible adaptation of multi-source heterogeneous data, and enhancing the channel expansion capability of data reading.

[0052] In one alternative approach, the steps of assigning a globally unique batch number to the insurance payment and receipt transaction data and using JDBC batch processing to write the insurance payment and receipt transaction data corresponding to the batch number into a temporary table include: A batch number generator is used to generate a globally unique batch number for the insurance payment and receipt transaction data, and a correspondence is established between the batch number and the insurance payment and receipt transaction data.

[0053] The batch number generator is a component responsible for generating globally unique batch numbers, typically based on algorithms such as timestamps, machine IDs, and serial numbers. For example, calling the batch number generator to generate batch number B20260101001 for the imported auto insurance premium data results in a number composed of the date 20260101 plus the first batch number of that day, 001. The mapping relationship refers to the association between the batch number and all insurance payment and receipt transaction data belonging to that batch. For example, establishing a mapping between batch number B20260101001 and 10,000 auto insurance premium data entries allows all subsequent operations to locate this batch of data using the batch number.

[0054] The insurance payment and receipt transaction data with the same batch number are divided into multiple data blocks according to the preset batch size.

[0055] The preset batch size refers to the number of data records committed per batch in JDBC batch processing, used to balance memory usage and network efficiency. For example, setting the preset batch size to 2000 records means that a batch commit is performed every 2000 records. A data block refers to multiple subsets into which a batch of data is divided according to the preset batch size. Each data block contains a fixed number of transaction records. For example, 10,000 car insurance premium records are divided into 5 data blocks, each containing 2000 records.

[0056] The insurance payment and collection transaction data in each data block is added to the batch processing buffer using the JDBC addBatch method, and the data in the batch processing buffer is written to a temporary table all at once using the executeBatch method when the preset batch processing size is reached.

[0057] The JDBC `addBatch` method refers to a method in the Java Database Connection interface used to add a single SQL statement to a batch processing block. For example, it iterates through 2000 records in each data block, calling the `addBatch` method on each record to add its insert statement to the batch buffer. The batch buffer is a memory area within the JDBC driver used to temporarily store SQL statements to be executed in batches. For instance, when `addBatch` is called to add records, the insert statement is temporarily stored in the batch buffer until the preset batch size is reached before execution.

[0058] The executeBatch method refers to a JDBC method used to send all SQL statements in a batch buffer to the database for execution at once. For example, after 2000 records have been added, the executeBatch method is called to submit these 2000 insert statements to the database for execution at once.

[0059] During the writing process, the writing status of each data block is recorded, and in the event of a writing error, the incomplete data blocks are resumed from where they left off based on the writing status and the batch number.

[0060] The write status refers to the success or failure marker for each data block during the database writing process, used for subsequent breakpoint resumption. For example, the write status of each data block is recorded; the first four data blocks are written successfully, and an exception occurs during the writing of the fifth data block, which is marked as failed. Breakpoint resumption means that when an exception occurs during batch writing, the write operation can be restarted from the failed data block based on the recorded write status, avoiding duplicate processing of completed data blocks. For example, after the fifth data block fails to write, based on the recorded write status and batch number B20260101001, the write operation is re-executed only on the fifth data block.

[0061] In the above optional methods, a batch number generator is used to establish the correspondence between batch numbers and transaction data. Data blocks are divided according to the preset batch size. The addBatch and executeBatch methods of JDBC are used to implement batch writing and record the writing status to support breakpoint resumption. This achieves high throughput data entry for massive amounts of data and breakpoint recovery in abnormal scenarios, enhancing the stability and continuity of data writing.

[0062] In one alternative approach, S2 specifically includes: After all the insurance payment and receipt transaction data is written to the temporary table, the set of verification rules corresponding to the insurance payment and receipt transaction data is read from the verification rule table. The verification rule table includes field names, verification types, and verification SQL templates.

[0063] The set of validation rules refers to a list of all validation rules applicable to a certain type of business data, read from the validation rule table. For example, the set of validation rules applicable to car insurance premium data read from the validation rule table includes two rules: validation of policy number not being empty and validation of premium amount being greater than 0.

[0064] Here, "field name" refers to the name of the field in the data table that needs to be validated, such as the policy number field or the premium amount field; for example, the field names in the validation rule set include policy_no and premium_amount. "Validation type" refers to the type of validation performed on the field, such as NOT NULL validation, range validation, or format validation; for example, the validation type for the policy_no field is NOT NULL validation, and the validation type for the premium_amount field is greater than 0 validation. "Validation SQL template" refers to a predefined framework used to generate the validation SQL statement, which includes placeholders for field names and validation conditions; for example, the validation SQL template is "SELECT COUNT(*) FROM temp_payment WHERE batch_id=? AND (${fieldName} IS NULL ${condition})", where the placeholders are replaced with the specific field names and validation conditions during generation.

[0065] The batch verification SQL statement is dynamically generated based on the batch number, the field name, the verification type, and the verification SQL template. The batch verification SQL statement is used to perform legality verification on all insurance payment and receipt transaction data of the batch number in a single database scan based on the field name and the verification type.

[0066] Among them, legality verification refers to the process of verifying whether insurance payment and receipt transaction data complies with business rules; for example, performing legality verification on each piece of auto insurance premium data with batch number B20260101001 to check whether the policy number is empty and whether the premium amount is greater than 0.

[0067] The batch verification SQL statement is sent to the database layer for execution. The database layer performs a one-time batch scan on the insurance payment and receipt transaction data with the batch number in the temporary table and returns a set of verification results.

[0068] The verification result set refers to the set of results returned by the database after executing the batch verification SQL statement, which usually includes the batch number and statistical information; for example, the verification result set returned by the database includes the batch number B20260101001 and the number of 10 erroneous records detected.

[0069] Based on the verification result set, generate the error code and error description corresponding to each insurance payment and receipt transaction data, and write the error code and error description back to the verification result field corresponding to each insurance payment and receipt transaction data in the temporary table in batches as verification results.

[0070] Error codes are codes used to identify specific types of verification failures, facilitating program identification and processing. For example, error code E001 is defined for an empty policy number, and error code E002 is defined for a premium amount less than or equal to 0. Error descriptions are human-readable textual explanations of the reasons for verification failures. For example, the error description for an empty policy number is "The policy number cannot be empty," and the error description for a premium amount less than or equal to 0 is "The premium amount must be greater than zero."

[0071] The verification result field refers to the field in the temporary table used to store the verification result of each data record, which usually includes the error code and error description. For example, the temporary table temp_payment has added two verification result fields, error_code and error_desc, to write back the verification result of each record.

[0072] In the above optional methods, the field names, verification types and verification SQL templates are further read from the verification rule table, and batch verification SQL statements are dynamically concatenated and generated. A one-time batch scan is then performed at the database layer, and the error codes and error descriptions are written back to the verification result field of the temporary table in batches. This realizes the dynamic configuration of the verification logic and the one-time batch verification at the database layer, reducing the overhead of multiple network round trips caused by verification one by one.

[0073] In one alternative approach, S3 specifically includes: After the verification results are written back to the temporary table in batches, it is determined whether all insurance payment and receipt transaction data with the batch number have passed the legality verification.

[0074] If all validations pass, a batch processing task is created based on the batch number, and the batch processing task is submitted to the distributed task scheduler.

[0075] The system resource load and task queue status are obtained through the distributed task scheduler. The system resource load includes CPU utilization and memory utilization, and the task queue status includes the number of tasks to be processed and the queuing time.

[0076] CPU utilization refers to the percentage of the execution node's central processing unit (CPU) in use, reflecting the workload of the node's computing resources. For example, if the distributed task scheduler finds that node A's CPU utilization is 75% and node B's is 30%, it will prioritize node B. Memory utilization refers to the percentage of the execution node's memory in use, reflecting the scarcity of the node's memory resources. For example, if the scheduler simultaneously checks that node B's memory utilization is 40% and node C's is 85%, then node B becomes the optimal choice for the target execution node.

[0077] The number of pending tasks refers to the number of tasks in the current queue of the distributed task scheduler that have not yet been assigned or completed. For example, if there are currently 15 pending tasks in the scheduler queue, a newly submitted task with batch number B20260101001 will wait in the queue. The queuing time refers to the estimated time a task will need to wait in the scheduler queue, typically estimated based on the number of pending tasks and the average processing time. For example, based on the current number of pending tasks and the average processing time, a new task is expected to wait in the queue for 30 seconds.

[0078] Based on the system resource load and the task queue status, a target execution node is dynamically selected from multiple candidate execution nodes, and the batch processing task is assigned to the target execution node.

[0079] In this context, candidate execution nodes refer to all available server instances in the distributed system capable of executing batch processing tasks; for example, there may be three candidate execution nodes A, B, and C in a distributed cluster. The target execution node refers to the node that is ultimately assigned to execute a specific batch processing task after being dynamically selected by the scheduler; for example, the scheduler dynamically selects node B from nodes A, B, and C as the target execution node for batch number B20260101001 based on system resource load and task queue status.

[0080] Through the target execution node, the insurance payment and receipt transaction data corresponding to the batch number is divided into multiple sub-tasks according to the preset sharding dimensions. Each sub-task corresponds to a data shard. The sharding dimensions include at least one of the following: institution dimension, product dimension, and date dimension.

[0081] In this context, a subtask refers to a smaller-granularity execution unit generated after the batch processing task is sharded. Each subtask is responsible for processing one data shard. For example, the batch processing task with batch number B20260101001 is split into three subtasks: Institution X, Institution Y, and Institution Z. A data shard refers to a subset of the entire batch data divided according to a sharding dimension. Each subset contains all transaction records under that dimension. For example, the data shard corresponding to the Institution X subtask contains all auto insurance premium records for all policyholders located at Institution X.

[0082] The organizational dimension refers to the standard for dividing data according to the internal organizational structure of the insurance company (such as branches and sub-branches). For example, when sharding data, the organizational dimension is used to split the data according to the policyholder's affiliated organization X, Y, and Z, allowing each organization to handle its own business independently. The product dimension refers to the standard for dividing data according to the type of insurance product (such as auto insurance, property insurance, and accident insurance). For example, if multiple products are being processed simultaneously, data can be sharded according to the product dimension, but in this example, only auto insurance is involved, so the organizational dimension is mainly used. The date dimension refers to the standard for dividing data according to a time range (such as by day or by month). For example, for large amounts of data spanning multiple months, data can be sharded according to the date dimension to process records from different months separately.

[0083] The multiple subtasks are distributed to multiple worker threads corresponding to the target execution node for parallel execution.

[0084] In this context, a worker thread refers to a thread within an execution node used to process subtasks in parallel, with each thread executing one subtask independently. For example, the target execution node may allocate a worker thread to each of the three subtasks, and the three threads may process their respective data shards simultaneously.

[0085] In the above optional methods, after the verification is passed, a batch processing task is created and submitted to the distributed task scheduler. The target execution node is dynamically selected based on CPU utilization, memory utilization and task queue status. Subtasks are divided by organization, product or date dimension and distributed to multiple worker threads for parallel execution. This realizes dynamic scheduling based on resource load and parallel processing of multi-dimensional data sharding, improving the balance and concurrency efficiency of task execution.

[0086] In one alternative approach, S4 specifically includes: When the multiple subtasks are executed in parallel, a subtask status record is created for each subtask, and the execution status changes of each subtask are captured in real time through an event listening mechanism.

[0087] The subtask status record refers to the creation of a status tracking object for each subtask, recording its current execution stage. For example, a subtask status record is created for organization X subtask, initially showing "pending," updating to "in execution" during execution, and updating to "completed" upon completion. The event listening mechanism refers to the method of capturing state changes in real time by listening to events emitted during subtask execution. For example, an event listener is registered; when the subtask status changes (such as start, complete, or fail), the listener receives a notification and triggers subsequent processing.

[0088] Among them, the execution state change refers to the process of a subtask changing from one state to another, such as changing from pending to executing; for example, when subtask X of organization starts to execute, the execution state changes from pending to executing, and the event listening mechanism captures this change.

[0089] Based on the changes in the execution status of all subtasks, the overall execution status corresponding to the batch number is aggregated and generated.

[0090] The overall execution status refers to the comprehensive status of the entire batch processing task, which is formed by aggregating the statuses of all subtasks, such as partially completed or fully completed. For example, when subtasks X and Y of organization are completed, while subtask Z of organization is still being executed, the overall execution status is partially completed.

[0091] The overall execution status is written in real time to the execution status field corresponding to the batch number in the task status table, and the task progress percentage is calculated based on the number of completed sub-tasks and the total number of sub-tasks.

[0092] The execution status field refers to the database column in the task status table used to store the overall execution status corresponding to the batch number; for example, the task status table has a field named status, which is used to store the overall execution status of batch number B20260101001.

[0093] The number of completed subtasks refers to the number of subtasks that have been successfully executed within a batch task; for example, if subtasks for organization X and organization Y have been completed, the number of completed subtasks is 2. The total number of subtasks refers to the total number of subtasks derived from a batch task; for example, the task with batch number B20260101001 is divided into 3 subtasks, and the total number of subtasks is 3.

[0094] The task progress percentage refers to the percentage of completed subtasks out of the total number of subtasks, which is used to visually represent the degree of task completion. For example, based on the number of completed subtasks 2 and the total number of subtasks 3, the task progress percentage is calculated to be 66.7%.

[0095] The task progress percentage and the overall execution status are pushed to the front end for display as task progress information through a pre-established asynchronous communication connection.

[0096] Asynchronous communication connection refers to a long-lived connection established with the front end for real-time message push, such as a WebSocket connection; for example, an asynchronous WebSocket communication connection is established with the operator's browser front end to push task progress information.

[0097] When any subtask fails, the overall execution status is updated to failure, and a failure notification is pushed to the front end via the asynchronous communication connection.

[0098] Among them, failure notification refers to the alarm information pushed to the front end when a task or subtask fails to execute; for example, when subtask Z of organization fails due to network timeout, a failure notification is pushed to the front end through an asynchronous communication connection to inform the operator that the subtask has failed.

[0099] In the above optional methods, an event listening mechanism is used to capture state changes when subtasks are executed in parallel, aggregate and generate the overall execution state and write it to the task status table in real time, calculate the task progress percentage and push it to the front end through an asynchronous communication connection, update the status and push failure notification when failure occurs, thereby realizing real-time aggregation of execution status and asynchronous push of task progress, improving the user's perception of the batch processing process and interactive experience.

[0100] In one alternative approach, S5 specifically includes: When any subtask fails, the failure type corresponding to the subtask is obtained, which is either a network timeout type, a data conflict type, or a system exception type.

[0101] Among these, network timeout type refers to failures caused by network connection timeouts or communication interruptions; for example, if subtask Z of organization waits for a response for more than 5 seconds when calling a remote service, it is judged as a network timeout failure. Data conflict type refers to transaction failures caused by concurrent data modification, unique key conflicts, etc.; for example, if a record is locked by another transaction or the same primary key already exists, it may cause a data conflict failure, but this did not occur in this example. System exception type refers to failures caused by internal failures such as program errors or database connection drops; for example, if a database service restart causes a connection loss, it is a system exception failure.

[0102] Based on the failure type, the corresponding retry threshold and retry interval are queried from the pre-configured retry policy table.

[0103] The retry policy table refers to a pre-configured table in the database used to store the retry threshold and retry interval for different failure types. For example, the retry policy table defines the retry threshold for the network timeout type as 3 times and the retry interval as 10 seconds.

[0104] The retry threshold refers to the maximum number of retries allowed for a specific failure type; for example, the retry threshold for network timeout is 3, meaning a maximum of 3 retries. The retry interval refers to the time interval between each retrieval, typically used to prevent immediate retrying from failing again; for example, the retry policy table specifies a 10-second retry interval for network timeout, meaning a 10-second wait after each retry before proceeding to the next.

[0105] Within the retry threshold range, the subtask is automatically retried according to the retry interval, and the retry count is updated after each retry.

[0106] The retry count refers to the number of times a failed task has been retried. For example, after the first retry of subtask Z, the retry count is updated to 1, and after the second retry, it is updated to 2.

[0107] If the subtask still fails to execute successfully after the retry count reaches the retry count threshold, the data fragment corresponding to the subtask is transferred to the dead letter queue as a whole.

[0108] Read the data fragments and failure types corresponding to the subtask from the dead-letter queue, generate the exception information containing the batch number, the subtask identifier and the reason for failure, and write the exception information into the exception record table.

[0109] The failure reason refers to a specific description of the cause of the task execution failure, such as a network connection timeout; for example, the failure reason recorded in the exception information is "timeout for remote service call, no response for 5 seconds". The exception log table refers to a table in the database used to persistently store exception information for subsequent auditing and troubleshooting; for example, exception information containing batch number B20260101001, subtask identifier, and failure reason is written to the exception log table exception_log.

[0110] In the above optional methods, when a subtask fails, the failure type (network timeout, data conflict, or system exception) is obtained, the retry policy table is queried to execute automatic retry, and after reaching the threshold, the retry is transferred to the dead letter queue. An exception information containing the batch number, subtask identifier, and failure reason is generated and written to the exception record table. This realizes the centralized management of differentiated retry policies and failed tasks, and improves the pertinence and traceability of exception handling.

[0111] In another embodiment of the intelligent elastic batch processing method for massive insurance payment and collection transactions of the present invention, the following steps are specifically included: S10. In response to the batch transaction data import request, read the property insurance payment and receipt transaction data to be processed from the multi-source data interface, assign a globally unique batch number to the property insurance payment and receipt transaction data, and use JDBC batch processing to write the property insurance payment and receipt transaction data corresponding to the batch number into a temporary table in batches; when reading data, dynamically match the corresponding data reading channel according to the data source identifier carried in the import request, and generate metadata tags containing data source type and collection timestamp after parsing the original transaction data, and store them together with the property insurance payment and receipt transaction data into the temporary table.

[0112] S20. After the data is written, based on the pre-configured verification rule table, batch verification SQL statements are dynamically generated according to the batch number. At the database layer, a one-time batch verification is performed on the property insurance payment and receipt transaction data with batch numbers in the temporary table, and the verification results are written back to the temporary table in batches. The verification rule table supports online hot updates. The verification rule table predefines field names, verification types, verification SQL templates, and dependencies between fields. When dynamically generating batch verification SQL statements, the verification order is automatically adjusted according to the dependencies to ensure that all legality verifications are completed in one database scan.

[0113] S30. After verification, a batch processing task is generated based on the batch number and submitted to the distributed task scheduler. The distributed task scheduler dynamically allocates execution nodes based on system resource load and task queue status to perform sharding processing on the property insurance payment and receipt transaction data corresponding to the batch number. During sharding processing, a sharding strategy library is pre-configured based on the business dimensions of property insurance. The sharding strategy library includes four strategies: sharding by insurance type code, sharding by regional institution code, sharding by policy effective date, and sharding by claim status. The distributed task scheduler automatically selects the optimal sharding strategy based on the statistical characteristics of the property insurance payment and receipt transaction data corresponding to the current batch number, and performs secondary splitting on shards with excessive data volume based on data distribution to avoid data skew.

[0114] S40. During the execution of the sharded processing task, the execution status corresponding to the batch number in the task status table is updated in real time, and the task progress information is pushed to the front end through an asynchronous communication mechanism. The task status table not only records the overall execution status, but also records the execution status, number of processing records and processing time of each data shard. When the asynchronous communication mechanism pushes the task progress information to the front end, it also pushes the detailed progress of each data shard, which supports the front end to drill down and view by shard dimension.

[0115] S50. When a fragmentation processing task fails, a retry mechanism is automatically triggered based on the failure type. After a preset number of retries, the fragmentation processing task is transferred to a dead-letter queue and the exception information is recorded. The retry mechanism adopts an exponential backoff strategy. Based on the failure type, the corresponding initial retry interval, maximum number of retries, and backoff factor are queried from the pre-configured retry strategy table. After each retry failure, the next retry interval is extended according to the backoff factor. When transferred to the dead-letter queue, the complete context information of the failed data fragment, including batch number, fragment identifier, failure type, retry count, and original data content, is encapsulated into exception information and written to the exception record table for manual analysis.

[0116] This embodiment uses JDBC batch processing to write property insurance payment and receipt transaction data into a temporary table and assign a globally unique batch number. It performs a one-time batch verification at the database layer, dynamically segments the data based on system resource load and task queue status, pushes task progress in real time, and manages anomalies using exponential backoff retries and dead-letter queues. This solves the technical problems of traditional scheduled batch operations, such as full page queries causing slow front-end rendering and heavy network transmission load, repeated database queries caused by multiple selections resulting in excessive I / O overhead, synchronous processing mechanisms forcing users to wait for a long time, and weak anomaly handling capabilities. It significantly improves the batch processing efficiency, system resource utilization, and anomaly recovery capabilities for massive insurance payment and receipt transactions, and optimizes the user's interactive experience in the batch processing process.

[0117] Figure 2 This diagram illustrates the structure of an embodiment of an intelligent, flexible batch processing system 200 for massive insurance payment and collection transactions provided by the present invention. Figure 2 As shown, the intelligent, flexible batch processing system 200 for massive insurance payment and collection transactions includes: The writing module 201 is used to respond to the import request of batch transaction data, read the insurance payment and receipt transaction data to be processed from the multi-source data interface, assign a globally unique batch number to the insurance payment and receipt transaction data, and use JDBC batch processing to write the insurance payment and receipt transaction data corresponding to the batch number into a temporary table in batches. The verification module 202 is used to dynamically generate batch verification SQL statements based on the batch number according to the pre-configured verification rule table after the data is written, perform a one-time batch verification on the insurance payment and receipt transaction data with the batch number in the temporary table at the database layer, and write the verification results back to the temporary table in batches. The allocation module 203 is used to generate a batch processing task based on the batch number after the verification is passed and submit it to the distributed task scheduler. The distributed task scheduler dynamically allocates execution nodes to perform fragmented processing of the insurance payment and receipt transaction data corresponding to the batch number based on the system resource load and task queue status. The update module 204 is used to update the execution status corresponding to the batch number in the task status table in real time during the execution of the segmented processing task, and push task progress information to the front end through an asynchronous communication mechanism. The detection module 205 is used to automatically trigger a retry mechanism according to the failure type when the fragmentation processing task is detected to have failed, and to transfer the fragmentation processing task to the dead letter queue and record the abnormal information after the retry reaches a preset number of times.

[0118] In one alternative embodiment, the writing module 201 is specifically used for: The system receives externally initiated import requests through a predefined data import interface. The import requests carry the data source identifier of the batch transaction data to be processed. The corresponding data reading channel is determined from the multi-source data interface based on the data source identifier. The multi-source data interface includes a file data source interface, a message queue data source interface, and a business database table data source interface. The original transaction data is read from the corresponding data source through the data reading channel, and the original transaction data is parsed into the insurance payment and receipt transaction data.

[0119] In one alternative embodiment, the writing module 201 is specifically used for: A batch number generator is used to generate a globally unique batch number for the insurance payment and receipt transaction data, and a correspondence is established between the batch number and the insurance payment and receipt transaction data. The insurance payment and receipt transaction data with the same batch number are divided into multiple data blocks according to the preset batch size; The insurance payment and collection transaction data in each data block is added to the batch processing buffer using the JDBC addBatch method, and the data in the batch processing buffer is written to a temporary table all at once using the executeBatch method when the preset batch processing size is reached. During the writing process, the writing status of each data block is recorded, and in the event of a writing error, the incomplete data blocks are resumed from where they left off based on the writing status and the batch number.

[0120] In one alternative embodiment, the verification module 202 is specifically used for: After all the insurance payment and receipt transaction data is written to the temporary table, the set of verification rules corresponding to the insurance payment and receipt transaction data is read from the verification rule table. The verification rule table includes field names, verification types, and verification SQL templates. Based on the batch number, the field name, the verification type, and the verification SQL template, the batch verification SQL statement is dynamically generated. The batch verification SQL statement is used to perform legality verification on all insurance payment and receipt transaction data of the batch number in a single database scan based on the field name and the verification type. The batch verification SQL statement is sent to the database layer for execution. The database layer performs a one-time batch scan on the insurance payment and receipt transaction data with the batch number in the temporary table and returns a set of verification results. Based on the verification result set, generate the error code and error description corresponding to each insurance payment and receipt transaction data, and write the error code and error description back to the verification result field corresponding to each insurance payment and receipt transaction data in the temporary table in batches as verification results.

[0121] In an alternative embodiment, the allocation module 203 is specifically used for: After the verification results are written back to the temporary table in batches, it is determined whether all insurance payment and receipt transaction data with the batch number have passed the legality verification. If all validations pass, a batch processing task is created based on the batch number, and the batch processing task is submitted to the distributed task scheduler. The system resource load and task queue status are obtained through the distributed task scheduler. The system resource load includes CPU utilization and memory utilization, and the task queue status includes the number of tasks to be processed and the queuing time. Based on the system resource load and the task queue status, a target execution node is dynamically selected from multiple candidate execution nodes, and the batch processing task is assigned to the target execution node; Through the target execution node, the insurance payment and receipt transaction data corresponding to the batch number is divided into multiple sub-tasks according to the preset sharding dimensions. Each sub-task corresponds to a data shard. The sharding dimensions include at least one of the following: institution dimension, product dimension, and date dimension. The multiple subtasks are distributed to multiple worker threads corresponding to the target execution node for parallel execution.

[0122] In an alternative embodiment, the update module 204 is specifically used for: When the multiple subtasks are executed in parallel, a subtask status record is created for each subtask, and the execution status changes of each subtask are captured in real time through an event listening mechanism. Based on the changes in the execution status of all subtasks, the overall execution status corresponding to the batch number is aggregated and generated; The overall execution status is written in real time to the execution status field corresponding to the batch number in the task status table, and the task progress percentage is calculated based on the number of completed sub-tasks and the total number of sub-tasks. The task progress percentage and the overall execution status are pushed to the front end for display as task progress information through a pre-established asynchronous communication connection. When any subtask fails, the overall execution status is updated to failure, and a failure notification is pushed to the front end via the asynchronous communication connection.

[0123] In an alternative embodiment, the detection module 205 is specifically used for: When any subtask fails to execute, the failure type corresponding to the subtask is obtained. The failure type is either a network timeout type, a data conflict type, or a system exception type. Based on the failure type, query the corresponding retry count threshold and retry interval from the pre-configured retry policy table; Within the retry threshold range, the subtask is automatically retried according to the retry interval, and the retry count is updated after each retry. When the retry count reaches the retry threshold and the subtask still fails to execute successfully, the data fragment corresponding to the subtask is transferred to the dead letter queue as a whole. Read the data fragments and failure types corresponding to the subtask from the dead-letter queue, generate the exception information containing the batch number, the subtask identifier and the reason for failure, and write the exception information into the exception record table.

[0124] It should be noted that the beneficial effects of the intelligent elastic batch processing system 200 for massive insurance payment and collection transactions provided in the above embodiments are the same as those of the intelligent elastic batch processing method for massive insurance payment and collection transactions described above, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0125] The intelligent flexible batch processing system 200 for massive insurance payment transactions of the present invention can be a computer program (including program code) running on a computer device. For example, the intelligent flexible batch processing system 200 for massive insurance payment transactions of the present invention is an application software that can be used to execute the corresponding steps in the intelligent flexible batch processing method for massive insurance payment transactions of the present invention.

[0126] In some embodiments, the intelligent flexible batch processing system 200 for massive insurance payment transactions of the present invention can be implemented in a combination of hardware and software. As an example, the intelligent flexible batch processing system 200 for massive insurance payment transactions of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the intelligent flexible batch processing method for massive insurance payment transactions of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0127] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0128] An electronic device according to an embodiment of the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-mentioned intelligent flexible batch processing methods for massive insurance payment and collection transactions. That is, an electronic device according to an embodiment of the present invention may include, but is not limited to: a processor and a memory; the memory is used to store the computer program; the processor is used to execute the intelligent flexible batch processing method for massive insurance payment and collection transactions shown in any embodiment of the present invention by calling the computer program.

[0129] In one alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3The illustrated electronic device 4000 includes a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 4004 is not limited to one type, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0130] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 4001 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0131] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 4002 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The bus 4002 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0132] The memory 4003 may be ROM (Read Only Memory) or other types of static storage devices capable of storing static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices capable of storing information and instructions, or EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0133] The memory 4003 stores application code (computer program) for executing the present invention, and its execution is controlled by the processor 4001. The processor 4001 executes the application code stored in the memory 4003 to implement the content shown in the foregoing method embodiments.

[0134] Among them, electronic devices can also be terminal devices. A terminal device can be any terminal device that can install applications and access web pages through applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0135] It should be noted that, Figure 3 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0136] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements any of the above-mentioned intelligent elastic batch processing methods for massive insurance payment and collection transactions.

[0137] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0138] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the aforementioned intelligent and flexible batch processing method for massive insurance payment and collection transactions.

[0139] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0141] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0142] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0143] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0144] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0145] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0146] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A method for intelligent flexible batch processing for mass transaction of insurance receipts and payments, characterized in that, include: In response to the request to import batch transaction data, the system reads the insurance payment and receipt transaction data to be processed from the multi-source data interface, assigns a globally unique batch number to the insurance payment and receipt transaction data, and uses JDBC batch processing to write the insurance payment and receipt transaction data corresponding to the batch number into a temporary table in batches. After the data is written, based on the pre-configured verification rule table, batch verification SQL statements are dynamically generated according to the batch number. At the database layer, a one-time batch verification is performed on the insurance payment and receipt transaction data with the batch number in the temporary table, and the verification results are written back to the temporary table in batches. After the verification is passed, a batch processing task is generated according to the batch number and submitted to the distributed task scheduler. The distributed task scheduler dynamically allocates execution nodes to perform fragmented processing of the insurance payment and receipt transaction data corresponding to the batch number based on the system resource load and task queue status. During the execution of the segmented processing task, the execution status corresponding to the batch number in the task status table is updated in real time, and task progress information is pushed to the front end through an asynchronous communication mechanism; When the fragmentation processing task is detected to have failed, a retry mechanism is automatically triggered according to the failure type. After a preset number of retries, the fragmentation processing task is transferred to a dead letter queue and the exception information is recorded.

2. The intelligent flexible batch processing method for insurance oriented mass transaction settlement according to claim 1, characterized in that, The steps for reading insurance payment and collection transaction data to be processed from a multi-source data interface in response to a request to import bulk transaction data include: The system receives externally initiated import requests through a predefined data import interface. The import requests carry the data source identifier of the batch transaction data to be processed. The corresponding data reading channel is determined from the multi-source data interface based on the data source identifier. The multi-source data interface includes a file data source interface, a message queue data source interface, and a business database table data source interface. The original transaction data is read from the corresponding data source through the data reading channel, and the original transaction data is parsed into the insurance payment and receipt transaction data.

3. The intelligent elastic batch processing method for massive insurance payment and collection transactions according to claim 2, characterized in that, The steps of assigning a globally unique batch number to the insurance payment and receipt transaction data and using JDBC batch processing to write the insurance payment and receipt transaction data corresponding to the batch number into a temporary table include: A batch number generator is used to generate a globally unique batch number for the insurance payment and receipt transaction data, and a correspondence is established between the batch number and the insurance payment and receipt transaction data. The insurance payment and receipt transaction data with the same batch number are divided into multiple data blocks according to the preset batch size; The insurance payment and collection transaction data in each data block is added to the batch processing buffer using the JDBC addBatch method, and the data in the batch processing buffer is written to a temporary table all at once using the executeBatch method when the preset batch processing size is reached. During the writing process, the writing status of each data block is recorded, and in the event of a writing error, the incomplete data blocks are resumed from where they left off based on the writing status and the batch number.

4. The intelligent elastic batch processing method for massive insurance payment and collection transactions according to claim 3, characterized in that, After the data writing is completed, based on the pre-configured verification rule table, batch verification SQL statements are dynamically generated according to the batch number. A one-time batch verification is performed on the insurance payment and receipt transaction data with the batch number in the temporary table at the database layer, and the verification results are written back to the temporary table in batches. This includes the following steps: After all the insurance payment and receipt transaction data is written to the temporary table, the set of verification rules corresponding to the insurance payment and receipt transaction data is read from the verification rule table. The verification rule table includes field names, verification types, and verification SQL templates. Based on the batch number, the field name, the verification type, and the verification SQL template, the batch verification SQL statement is dynamically generated. The batch verification SQL statement is used to perform legality verification on all insurance payment and receipt transaction data of the batch number in a single database scan based on the field name and the verification type. The batch verification SQL statement is sent to the database layer for execution. The database layer performs a one-time batch scan on the insurance payment and receipt transaction data with the batch number in the temporary table and returns a set of verification results. Based on the verification result set, generate the error code and error description corresponding to each insurance payment and receipt transaction data, and write the error code and error description back to the verification result field corresponding to each insurance payment and receipt transaction data in the temporary table in batches as verification results.

5. The intelligent elastic batch processing method for massive insurance payment and collection transactions according to claim 4, characterized in that, After successful verification, a batch processing task is generated based on the batch number and submitted to the distributed task scheduler. The distributed task scheduler then dynamically allocates execution nodes based on system resource load and task queue status to perform fragmented processing of the insurance payment and receipt transaction data corresponding to the batch number. This includes the following steps: After the verification results are written back to the temporary table in batches, it is determined whether all insurance payment and receipt transaction data with the batch number have passed the legality verification. If all validations pass, a batch processing task is created based on the batch number, and the batch processing task is submitted to the distributed task scheduler. The system resource load and task queue status are obtained through the distributed task scheduler. The system resource load includes CPU utilization and memory utilization, and the task queue status includes the number of tasks to be processed and the queuing time. Based on the system resource load and the task queue status, a target execution node is dynamically selected from multiple candidate execution nodes, and the batch processing task is assigned to the target execution node; Through the target execution node, the insurance payment and receipt transaction data corresponding to the batch number is divided into multiple sub-tasks according to the preset sharding dimensions. Each sub-task corresponds to a data shard. The sharding dimensions include at least one of the following: institution dimension, product dimension, and date dimension. The multiple subtasks are distributed to multiple worker threads corresponding to the target execution node for parallel execution.

6. The intelligent elastic batch processing method for massive insurance payment and collection transactions according to claim 5, characterized in that, The steps involved in updating the execution status corresponding to the batch number in the task status table in real time during the execution of the segmented processing task, and pushing task progress information to the front end through an asynchronous communication mechanism, include: When the multiple subtasks are executed in parallel, a subtask status record is created for each subtask, and the execution status changes of each subtask are captured in real time through an event listening mechanism. Based on the changes in the execution status of all subtasks, the overall execution status corresponding to the batch number is aggregated and generated; The overall execution status is written in real time to the execution status field corresponding to the batch number in the task status table, and the task progress percentage is calculated based on the number of completed sub-tasks and the total number of sub-tasks. The task progress percentage and the overall execution status are pushed to the front end for display as task progress information through a pre-established asynchronous communication connection. When any subtask fails, the overall execution status is updated to failure, and a failure notification is pushed to the front end via the asynchronous communication connection.

7. The intelligent elastic batch processing method for massive insurance payment and collection transactions according to claim 6, characterized in that, When a failure is detected in the sharding processing task, a retry mechanism is automatically triggered based on the failure type, and after a preset number of retries, the sharding processing task is transferred to a dead-letter queue and the exception information is recorded. This includes the following steps: When any subtask fails to execute, the failure type corresponding to the subtask is obtained. The failure type is either a network timeout type, a data conflict type, or a system exception type. Based on the failure type, query the corresponding retry count threshold and retry interval from the pre-configured retry policy table; Within the retry threshold range, the subtask is automatically retried according to the retry interval, and the retry count is updated after each retry. When the retry count reaches the retry threshold and the subtask still fails to execute successfully, the data fragment corresponding to the subtask is transferred to the dead letter queue as a whole. Read the data fragments and failure types corresponding to the subtask from the dead letter queue, generate the exception information containing the batch number, the subtask identifier and the reason for failure, and write the exception information into the exception record table.

8. An intelligent, flexible batch processing system for massive insurance payment and collection transactions, characterized in that: include: The write module is used to respond to the import request of batch transaction data, read the insurance payment and receipt transaction data to be processed from the multi-source data interface, assign a globally unique batch number to the insurance payment and receipt transaction data, and use JDBC batch processing to write the insurance payment and receipt transaction data corresponding to the batch number into a temporary table in batches. The verification module is used to dynamically generate batch verification SQL statements based on the batch number according to the pre-configured verification rule table after the data is written, perform a one-time batch verification on the insurance payment and receipt transaction data with the batch number in the temporary table at the database layer, and write the verification results back to the temporary table in batches. The allocation module is used to generate batch processing tasks based on the batch number after the verification is passed and submit them to the distributed task scheduler. The distributed task scheduler dynamically allocates execution nodes to perform fragmented processing of the insurance payment and receipt transaction data corresponding to the batch number based on the system resource load and task queue status. The update module is used to update the execution status corresponding to the batch number in the task status table in real time during the execution of the segmented processing task, and push task progress information to the front end through an asynchronous communication mechanism. The detection module is used to automatically trigger a retry mechanism according to the failure type when the fragmentation processing task is detected to have failed, and to transfer the fragmentation processing task to the dead letter queue and record the abnormal information after the retry reaches a preset number of times.

9. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the intelligent elastic batch processing method for massive insurance payment transactions as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, implements the intelligent elastic batch processing method for massive insurance payment and collection transactions as described in any one of claims 1 to 7.

Images