Data processing method and device, equipment and storage medium

Abstract

The invention belongs to the technical field of data processing, is applied to the field of financial science and technology and the field of medical health, and discloses a data processing method, device and equipment and a storage medium, and the method comprises the steps: obtaining original business data, and storing the original business data in a preset distributed database; extracting basic data of the original business data and corresponding association proportion parameters from the preset distributed database based on a preset big data platform; based on a preset proportional relation, carrying out parallel calculation on the basic data and the associated proportional parameter to generate a data processing result; and writing the data processing result back to a specified data table of the preset distributed database. The invention provides a data processing method and device, equipment and a storage medium. The problems that in the prior art, a traditional processing method adopted during data processing is low in processing efficiency and the accuracy is difficult to guarantee are solved.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology and is applied in the fields of financial technology and healthcare. In particular, it relates to a data processing method, apparatus, device, and storage medium. Background Technology

[0002] In the fields of finance, insurance, and healthcare, the processing of large-scale business data is a core component supporting business decision-making and process automation. Taking reinsurance business in the insurance industry and health data statistics in the healthcare industry as examples, these scenarios typically require the extraction, calculation, and storage of massive amounts of structured data (such as policy information, premium amounts, and user health indicators). The core requirement lies in generating accurate business results (such as reinsurance premiums and health risk assessment coefficients) based on the proportional conversion of basic data and related parameters.

[0003] Traditional data processing methods suffer from the following technical drawbacks: Traditional systems often rely on centralized databases such as Oracle, using stored procedures or single-threaded scripts to perform data calculations. When faced with high-concurrency business requests (such as tens of thousands of policy change records per day) or large-scale data (such as health data of tens of millions of users), the high coupling between computing and storage resources in centralized architectures easily leads to problems such as computing task congestion and data read / write latency, resulting in a significant decrease in processing efficiency and an inability to meet the real-time requirements of business operations (such as insurance underwriting requiring results within minutes). In traditional methods, data extraction (such as retrieving basic data from database tables) and calculation logic (such as ratio conversion formulas) are often hard-coded together. When business rules change (such as adjusting ratio parameters), code modification and redeployment are required, resulting in extremely poor flexibility and a high risk of calculation errors, especially when processing large-scale data, where calculation accuracy is difficult to guarantee. Furthermore, the extraction of multi-source data relies on complex table join operations, which are prone to failure due to changes in data structure, increasing system maintenance costs. Traditional systems use a serial processing mode, and when the data volume reaches hundreds of thousands or more, serial processing causes the cumulative task time to increase linearly.

[0004] In summary, existing technologies employing traditional data processing methods suffer from low processing efficiency and difficulty in guaranteeing accuracy. Summary of the Invention

[0005] This invention provides a data processing method, apparatus, device, and storage medium to solve the problems of inaccurate underwriting materials and low efficiency in the traditional underwriting process used in the prior art.

[0006] In a first aspect, the present invention provides a data processing method, comprising:

[0007] Obtain the original business data and store the original business data in a preset distributed database;

[0008] Based on a pre-defined big data platform, the basic data and corresponding correlation ratio parameters of the original business data are extracted from the pre-defined distributed database.

[0009] Based on a preset proportional relationship, the basic data and the associated proportional parameters are calculated in parallel to generate data processing results;

[0010] The data processing results are written back to a designated data table in the preset distributed database.

[0011] In a second aspect, the present invention provides a data processing apparatus, comprising:

[0012] The acquisition module is used to acquire raw business data and store the raw business data in a preset distributed database.

[0013] The extraction module is used to extract the basic data and corresponding correlation ratio parameters of the original business data from the preset distributed database based on the preset big data platform;

[0014] The generation module is used to generate data processing results by parallel calculation of the basic data and the associated proportional parameters based on a preset proportional relationship.

[0015] The write-back module writes the data processing results back to a specified data table in the preset distributed database.

[0016] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described data processing method.

[0017] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described data processing method.

[0018] In the aforementioned data processing methods, devices, equipment, and storage media, a pre-set distributed database enables centralized storage of raw business data. Compared to the traditional model of scattered storage across multiple independent systems, this significantly improves the convenience and completeness of data acquisition. Extraction and parallel computation of raw business data effectively integrates basic data and related proportional parameters, rapidly generating processing results and forming a high-quality, standardized business dataset. This avoids decision-making biases caused by data dispersion and computational lag, laying a solid foundation for subsequent accurate analysis. Leveraging a pre-set big data platform and proportional relationship formulas to deeply mine the value of business data, key information and correlation patterns can be accurately extracted. Compared to traditional simple analysis methods based on single data, this provides more comprehensive and accurate support for business decisions, and the generated data processing results are more aligned with actual business needs, helping industry players to anticipate business dynamics. Combining the data processing results with pre-set business rules enables automated and intelligent business management. This method breaks through the limitations of traditional data processing applications in the fintech and healthcare fields. In the fintech field, it helps financial institutions optimize risk control and improve operational efficiency, while in the healthcare field, it provides medical institutions with precise diagnosis and treatment support and optimizes resource allocation. It achieves a dual improvement in the quality and efficiency of data processing in cross-domain applications, providing new impetus for industry development.

[0019] In summary, this solution can solve the problems of low processing efficiency and difficulty in guaranteeing accuracy in traditional data processing methods used in existing technologies. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating a data processing method according to an embodiment of the present invention;

[0022] Figure 2 yes Figure 1 A flowchart of step S120;

[0023] Figure 3 yes Figure 1 A flowchart of step S130;

[0024] Figure 4 This is a schematic diagram of a data processing device in one embodiment of the present invention;

[0025] Figure 5This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0026] Figure 6 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Please see Figure 1 As shown in the flowchart, an embodiment of the present invention provides a data processing method, which includes the following steps.

[0029] Step S110: Obtain the original business data and store the original business data in a preset distributed database.

[0030] It should be noted that the raw business data storage mechanism in step S110 is based on standardized processing technology using multi-source data fusion and a distributed architecture. Its core lies in achieving high-quality storage of business data through full-process control of extraction, transformation, and verification. This adapts to scenarios such as transaction data archiving and customer information management in the fintech field, as well as scenarios such as medical record storage and medical insurance data management in the healthcare field. The raw business data is multi-dimensional data reflecting business operations and user interactions across various industries. Its type and acquisition method are closely related to the characteristics of the business scenario and highly aligned with the business needs of the fintech and healthcare fields. At the data type level, it covers financial transaction data, such as transaction amount, transaction time, transaction type, and counterparty information; customer information data, including basic customer information (name, gender, age, contact information), occupation information, income status, and credit history; medical service data, involving medical records, examination reports, medication information, and hospitalization duration; and equipment operation data, such as the operating status, fault records, and maintenance logs of medical equipment. In the financial sector, raw business data can include policy information, transaction records, etc.; in the medical sector, it can cover patient medical records, treatment costs, etc. After standardization, this data lays the foundation for subsequent data analysis and business decisions.

[0031] In terms of storage devices and methods, it supports various types of pre-defined distributed databases, such as the Hadoop Distributed File System (HDFS), Apache Cassandra, and MongoDB as data storage terminals. For example, HDFS boasts high fault tolerance and high throughput, making it suitable for storing massive amounts of financial transaction logs and medical image data. Apache Cassandra, on the other hand, excels in high availability and linear scalability, and can be used to store frequently updated customer credit information and real-time medical data. In the fintech field, the data stored in these databases is used for risk assessment, fraud detection, and customer profiling, providing data support for financial institutions' business decisions. In the healthcare field, it can be used for electronic medical record management, medical resource allocation, and disease trend analysis, ensuring efficient operation and precision medicine in medical institutions.

[0032] Storage frequency can be flexibly set based on business scenarios and data generation speed. For example, high-frequency financial transaction data, which generates tens of thousands of records per second, needs to be written to a distributed database in real time; non-real-time updated data such as medical examination reports can be stored in batches by hour or day; customer information data is updated and stored promptly when changes occur. In addition, the acquired raw business data is stored in standardized formats, such as columnar storage formats like Parquet and ORC, or text formats like JSON and CSV. Data transmission and processing are carried out through distributed computing frameworks (such as Spark and Flink) to ensure that data can be securely stored, efficiently accessed, and deeply analyzed in both the fintech and healthcare fields.

[0033] In some embodiments of the present invention, the following steps are included before step S110:

[0034] Step S101: Extract the original business data from the business source system;

[0035] Step S102: Perform standardized data transformation on the extracted raw business data to generate structured data;

[0036] Step S103: After verifying the integrity and consistency of the structured data, the distributed storage process is triggered.

[0037] Specifically, in step S101, data extraction employs a hybrid strategy of "real-time incremental + batch full extraction." In the fintech field, real-time transaction data (e.g., 3000 payment records per second) from the core system is captured via direct database connection (e.g., Oracle's GoldenGate), while ETL tools (e.g., Informatica) extract the previous day's full policy data daily at midnight to ensure data integrity. For example, when a brokerage extracts users' stock trading records from the trading system, a real-time channel captures the instantaneous transaction data, while a batch channel supplements the data with the summary of holdings after overnight clearing, with dual-link coverage to avoid omissions. In the healthcare field, data is extracted from HIS and LIS systems based on medical data exchange standards (e.g., HL7v2.x): dynamic data such as outpatient prescriptions and test results are received in real-time through message middleware (e.g., Kafka), while static data such as inpatient medical records are extracted through scheduled tasks (e.g., at 2 AM daily). For example, when a hospital extracts blood routine data from the LIS system, it obtains the completed test results in real-time and batch extracts historically archived abnormal test records to ensure the timeliness and traceability of clinical data.

[0038] Step S102: Perform standardized data transformation processing on the extracted raw business data.

[0039] Specifically, in step S102, the conversion process combines domain-specific standards to achieve data structure reconstruction. In the fintech field, data is cleaned according to the "Financial Industry Data Security Specification": semi-structured PDF contracts are processed using OCR+NLP to extract fields such as "Party A Name" and "Signing Date," with the date format standardized to "yyyy-MM-dd," and the monetary unit converted to "RMB Yuan." Different expressions for the same entity, such as "Zhang San" and "Mr. Zhang," are normalized. For example, loan contract data from different branches are converted into a standardized JSON structure containing "Borrower ID, Loan Amount, and Interest Rate" to eliminate format differences. In the healthcare field, conversion follows medical standards such as SNOMEDCT and ICD-10: entities such as "Diagnosis Name" and "Medication Dosage" are extracted from medical record text using NLP tools, "High Blood Pressure" is mapped to the ICD-10 code "I10," units such as "mg" and "gram" are standardized to "g," and expressions such as "Bed Number 3" and "Bed 3" are standardized. For example, the phrase "fever 38.5 degrees, ibuprofen prescribed" in a handwritten medical record can be converted into structured data containing "symptoms: fever (R50.9), body temperature: 38.5℃, medication: ibuprofen (B01AC01)".

[0040] Specifically, in step S103, the verification process employs a multi-dimensional rule engine. In the fintech field, integrity verification checks whether required fields such as "transaction amount" and "transaction time" are missing. Consistency verification ensures the correctness of data logic through verification formulas (such as "total principal and interest = principal + principal × interest rate × term"). It also reconciles the "consumption amount" and "merchant receipt amount" in the credit card statement, marking any discrepancies exceeding 0.01 yuan as abnormal. After successful verification, an HBase Put operation is triggered, storing data in shards based on "user ID + timestamp". In the healthcare field, integrity verification checks whether core fields such as "patient ID" and "diagnosis result" are complete. Consistency verification verifies the reasonableness of data using medical rules (such as "when blood glucose is measured in mmol / L, the normal range is 3.9-6.1"). It verifies the correlation between the "admission diagnosis" and "discharge diagnosis" for the same patient, triggering manual review if logical conflicts occur. After successful verification, a MongoDB insert operation is triggered, storing data in shards based on "department + visit date," and simultaneously generating a verification report recording the field mapping relationship before and after the conversion.

[0041] Understandably, a preprocessing workflow that first extracts raw business data, then performs standardized transformation, and finally verifies it before triggering storage, can more accurately improve data quality compared to directly storing raw data in a database, laying a reliable foundation for subsequent distributed storage. This process is particularly crucial in the fintech field because accurately processing policy data, transaction logs, and other information helps financial institutions eliminate data format differences and quality issues, providing high-quality data support for risk assessment and compliance audits. This three-tiered collaborative preprocessing framework demonstrates strong robustness to various types of financial data. Whether it's high-frequency transaction records, complex contract texts, or customer information from multiple channels, as long as it belongs to financial business data, it can be transformed into standardized data through this workflow, thereby achieving unified data management and efficient application. Unlike traditional preprocessing that only performs simple format conversion, this method's multi-dimensional verification and deep standardization significantly improve the consistency and usability of financial data. After preprocessing, it forms an important foundation for the efficient operation of subsequent financial databases and the accurate output of data analysis models. For example, in a bank's customer relationship management system, preprocessed standardized data can be used to quickly integrate multi-dimensional customer information, generating customer profiles that meet business needs. This improves the efficiency of financial institutions' precision marketing and customer service quality, effectively meeting the financial industry's demand for high-quality preprocessing and in-depth application of business data. In the healthcare field, this three-stage collaborative preprocessing workflow can accurately process patient medical records, test reports, and other data, helping medical institutions eliminate data silos and format chaos, providing reliable data support for clinical diagnosis and medical research. This three-stage collaborative preprocessing framework is robust to various types of medical data. Whether it's real-time vital signs data, imaging reports, or handwritten medical records, as long as it belongs to medical business data, it can be transformed into standardized data through this workflow, thereby achieving the integration, sharing, and in-depth mining of medical data. Unlike traditional preprocessing that only performs simple format conversion, this method's multi-dimensional verification and deep standardization significantly improve the integrity and standardization of medical data. After preprocessing, it provides important support for the efficient operation of subsequent medical databases and the precise application of intelligent medical systems. For example, in a hospital's telemedicine platform, a patient's complete medical records can be quickly retrieved based on pre-processed standardized data, assisting doctors in making accurate diagnoses, generating treatment plans that comply with medical standards, improving the efficiency of medical institutions' diagnosis and treatment and their ability to collaborate on medical data across regions, and effectively meeting the healthcare sector's demand for high-quality pre-processing and efficient application of business data.

[0042] In some embodiments of the present invention, step S103 specifically includes:

[0043] Step 1031: Perform fragmentation processing on the structured data according to the business time characteristics;

[0044] Step 1032: Establish an independent transaction log recording mechanism for each data shard;

[0045] Step 1033: Verify cross-shard data consistency using a distributed transaction protocol.

[0046] It should be noted that the three sub-steps of step S103 form a chain of integrity and consistency assurance before the structured data is distributed and stored. Through business time-oriented sharding strategies, fine-grained logging, and cross-shard consistency verification, the reliability of data in a distributed environment is ensured, which is suitable for the high requirements of data traceability and accuracy in the financial and medical fields.

[0047] Specifically, in step 1031, sharding is used to logically divide data based on the business time dimension, balancing query efficiency and business relevance. In the fintech field, business time characteristics can be refined into transaction occurrence time, policy effective date, settlement cycle, etc. For example, bank transaction data is sharded at three levels: first-level sharding by year (e.g., 2023, 2024), second-level sharding by quarter, and third-level sharding by calendar month. All transfer transaction data for the first quarter of 2024 is stored in the corresponding shard node, and data from high-frequency transaction periods (e.g., weekdays 9:00-17:00) is additionally sharded at the hour level to improve real-time query speed. For policy data, sharding is done by "effective year + coverage period," such as policies effective in 2024 with a 20-year coverage period being stored centrally for easy renewal reminders and claims calculation. In the healthcare field, business time characteristics include consultation time, test report generation time, hospitalization period, etc. For example, hospital medical data is segmented by "year of visit + department + month": outpatient data for the cardiology department in March 2024 is stored in a dedicated segment, while emergency data, due to its high timeliness requirements, is further divided into sub-segments by "hour" (e.g., emergency records from 14:00 to 15:00 on March 15th). For long-term follow-up data of patients with chronic diseases, segmentation is done by "year + quarter," making it easier for doctors to track changes in the patient's condition at different stages. For example, blood glucose monitoring data for diabetic patients in the fourth quarter of 2023 can be quickly located and retrieved.

[0048] Specifically, in step 1032, the transaction log records cover the entire lifecycle of data operations, providing a traceable operation trajectory for each shard. In the fintech field, the log content includes the operation type (such as insert, update, delete), operation timestamp, operator ID, field values ​​before and after data change, and verification hash value. For example, the transaction log of a certain insurance policy shard will record in detail: "2024-05-20 10:15:30, operator ID 001, updated the premium amount of policy P20240056 from 5000 yuan to 5500 yuan, hash value before change xxx, hash value after change yyy". The log adopts the WAL (Write-Ahead Logging) mechanism, writing to the log before performing data updates, and the log of each shard is stored separately on a high-performance SSD, supporting fast retrieval by "shard ID + time range", meeting the compliance requirements of financial audit for operation traceability. In the healthcare field, transaction logs, in addition to recording basic operational information, must also be linked to compliance identifiers for medical operations, such as the doctor's employee number executing the prescription and the review status of the test report. For example, the log record for a certain test data segment might be: "2024-05-20 14:30:20, Operation Type: Add, Test Order Number L202401234, Tester ID: DrLi, Review Status: Approved, Data Field: Blood Glucose Value 5.6mmol / L, Log Signature: DrLi Digital Certificate". The logs are stored in a blockchain-style chain, with each log block containing the hash value of the previous block to prevent tampering, ensure the judicial traceability of medical data, and comply with the requirements for the authenticity of medical records in the "Regulations on the Prevention and Handling of Medical Disputes".

[0049] Specifically, in step 1033, cross-shard consistency verification achieves global consistency of data logic through a mature distributed transaction protocol. In the fintech field, the TCC (Try-Confirm-Cancel) protocol is used to handle cross-shard transactions: for example, when a user transfers money across banks, the shard where the sending account is located (Bank A) and the shard where the receiving account is located (Bank B) need to update their balances simultaneously. The Try phase freezes the transfer amount in both accounts, the Confirm phase actually deducts and adds funds, and the Cancel phase unfreezes funds in case of an anomaly. Simultaneously, a two-phase commit (2PC) protocol is used for final confirmation. The coordinator node sends a "ready to commit" request to all participating shards. After all shards return "ready to commit," a "confirm commit" instruction is sent, ensuring that the operations of deducting 1000 yuan from shard A and adding 1000 yuan to shard B either succeed simultaneously or fail simultaneously, avoiding discrepancies between the recorded and actual funds. In the healthcare field, the Saga protocol is used to handle cross-shard medical data updates: for example, when a patient is transferred to another hospital, the original hospital's medical record shard and the new hospital's receiving shard need to update the patient's status synchronously. The first phase involves the original hospital segment executing the "marking patient transfer out" operation. The second phase involves the new hospital segment executing the "receiving patient and associating historical records" operation. If the second phase fails, a compensation mechanism is triggered—the original hospital segment revokes the "transfer out" mark and restores the patient's in-hospital status. Simultaneously, a scheduled task compares the associated fields (such as patient ID and medical card number) across different segments to ensure consistency in key information between the original hospital's "discharge diagnosis" and the new hospital's "admission diagnosis," thus preventing gaps in medical data.

[0050] Understandably, by sharding data based on business time characteristics, establishing independent transaction logs, and employing distributed transaction protocols for verification, this approach more accurately ensures the integrity and consistency of structured data in distributed storage compared to simple random sharding based on data volume or methods lacking transaction records. This mechanism is particularly crucial in the fintech field because accurately sharding data according to business time (such as transaction dates or policy effective dates), establishing detailed transaction logs, and verifying cross-shard consistency helps financial institutions efficiently manage massive amounts of transaction data, providing a traceable and complete data chain for fund clearing, compliance audits, and more. This sharding and transaction management framework based on business time characteristics demonstrates strong robustness to various types of financial data. Whether it's high-frequency real-time transaction data, periodic policy data, or cross-institutional settlement data, as long as it follows the time logic of financial business, this mechanism can achieve orderly data storage and consistency verification, thereby ensuring the reliability of distributed storage. Unlike traditional sharding methods that only focus on balanced data storage, this method's transaction log recording and cross-shard verification features significantly improve the traceability and fault tolerance of financial data. After data sharding and consistency verification are completed, a crucial foundation is laid for rapid querying of subsequent financial data and accurate location of abnormal transactions. For example, in a bank's cross-border payment system, the entire process of a cross-border transaction can be quickly traced based on the sharding structure and transaction logs according to transaction time, generating an audit report that meets international clearing standards. This improves the risk management level of financial institutions and the efficiency of cross-system data collaboration, effectively meeting the high requirements of the financial industry for data integrity and consistency in distributed storage. In the healthcare field, this mechanism can accurately shard medical data according to business time (such as consultation time, test report generation time), establish detailed transaction logs, and verify cross-shard consistency, helping medical institutions reliably store massive amounts of medical data and providing complete and traceable data support for clinical diagnosis and treatment, medical quality assessment, and more. This sharding and transaction management framework based on business time characteristics is highly robust to complex medical data. Whether it is real-time vital sign data, periodically generated physical examination reports, or cross-departmental patient treatment data, as long as the medical business time logic is followed, this mechanism can achieve orderly storage and consistency verification of data, thereby ensuring the reliability of distributed storage. Unlike traditional sharding methods that only focus on balanced data storage, this approach significantly improves the continuity and traceability of medical data through transaction log recording and cross-shard verification. After data sharding and consistency verification are completed, it provides crucial support for efficient access to subsequent medical data and complete retrospective analysis of patient treatment processes. For example, in a regional healthcare information platform, based on a sharding structure by visit time and transaction logs, patient records from different hospitals can be quickly integrated to generate complete electronic health records that meet clinical needs. This enhances the collaborative capabilities of medical institutions and the efficiency of cross-regional medical data sharing, effectively meeting the high requirements of the healthcare field for data integrity and consistency in distributed storage.

[0051] In some embodiments of the present invention, step S103 is followed by the following steps:

[0052] Step S104: Monitor the execution status of the data processing task;

[0053] Step S105: Perform automatic recovery processing on abnormal tasks, wherein the automatic recovery processing includes marking the abnormal state and adding it to the task retry queue;

[0054] Step S106: When the number of retries reaches a preset threshold, the exception handling process is triggered.

[0055] Step S107: After manually intervening in the abnormal data through the abnormal handling process management interface, the calculation task is re-executed.

[0056] It should be noted that steps S104 to S107 constitute the full-link anomaly management system for data processing tasks. Through a collaborative mechanism of real-time monitoring, automatic recovery, threshold triggering, and manual intervention, it ensures that data processing tasks can respond in an orderly manner when encountering anomalies, adapting to the high requirements of the financial technology field for the continuity of transaction data processing, as well as the stringent standards of the medical and health field for the accuracy of medical data processing.

[0057] Specifically, in step S104, the status monitoring adopts a multi-dimensional indicator real-time collection and visualization strategy. In the fintech field, monitoring indicators cover task progress (e.g., "80% complete"), processing time (e.g., "Current task has been running for 120 seconds, estimated remaining 30 seconds"), resource usage (e.g., CPU utilization 75%, memory usage 60%), and data verification results (e.g., "100,000 data entries processed, 23 outliers"). Status data is updated every second using tools such as Zabbix or Grafana. For high-frequency transaction processing tasks (e.g., 5,000 payment settlements per second), millisecond-level monitoring granularity is set. When a task experiences "stagnation exceeding 30 seconds" or "data verification anomaly rate exceeding 1%", it is automatically marked as "suspected anomaly" and an alert is triggered. For example, in a bank's real-time transfer data processing task, if only 30,000 out of a batch of 50,000 transaction data are processed within one minute, the system immediately switches the task status from "normal" to "pending verification". In the healthcare field, monitoring indicators include task start time, current processing stage (e.g., "test data format conversion in progress"), data integrity (e.g., "100 medical records received, 3 missing key fields"), and interaction status with the medical system (e.g., "normal connection with the electronic medical record system"). Tasks are tracked in real time through a dedicated medical monitoring platform (e.g., the OpenMRS monitoring module). For emergency data processing tasks (e.g., real-time analysis of ICU patient vital signs), a rule is set to issue an alert if the processing delay exceeds 10 seconds. For routine physical examination data batch processing tasks, the rule is relaxed to an alert if the delay exceeds 5 minutes. For example, in the hospital's automatic test report entry task, if 20 newly received blood routine reports are not processed within 10 consecutive minutes, the system marks the task as "abnormal and pending."

[0058] Specifically, in step S105, the automatic recovery process achieves rapid restart of abnormal tasks through status marking and queue scheduling. In the fintech field, when a transaction data aggregation task is marked as abnormal due to "database connection timeout," the system automatically executes the following: Recording "Abnormal type: connection interruption, occurrence time: 2025-07-10 09:15:30" in the task log; adding the task ID (e.g., "T20250710001") to the retry queue, which is sorted by "task priority + abnormal occurrence time," assigning priority 1 (highest) to VIP customer policy calculation tasks and priority 3 to ordinary transfer tasks. The retry queue adopts a FIFO (first-in, first-out) mechanism, while setting a "retry after 10 seconds" delay strategy for temporary abnormal tasks such as "network fluctuations," and setting "immediate retry" for structural abnormal tasks such as "data format errors" to quickly expose deeper problems. For example, a batch of credit card bill calculation tasks is marked as abnormal due to a network interruption and is retrieved from the queue for retry after 5 seconds. In the healthcare field, when a patient's medical record structure conversion task encounters an error due to "PDF format corruption," the system performs the following actions: Marks the exception status as "format error" and associates it with the error log (e.g., "Page 5 cannot be parsed, error code: PDF-003"); adds the task to a retry queue, prioritizing it according to "Emergency > Outpatient > Inpatient," allowing emergency patients' medical record processing tasks to be moved to the head of the queue. For exceptions caused by "temporary disk I / O busy," a "retry after 30 seconds" setting is implemented; for exceptions caused by "blurred medical record content," a "retry after 5 minutes" setting is implemented (waiting for OCR recognition service resources to be released). For example, if a patient's progress note conversion task encounters a disk busy exception, it automatically retryes after 30 seconds and completes successfully.

[0059] Specifically, in step S106, the threshold setting is dynamically adjusted based on task type and anomaly cause, and the triggering process includes multi-channel alarms and resource isolation operations. In the fintech field, the preset thresholds are graded according to task importance: the retry threshold for core transaction settlement tasks is 3 times, and for ordinary data backup tasks it is 5 times. When a fund net asset value calculation task fails to retry 3 times consecutively due to "data source file lock", the system immediately: sends an alarm to the operations team via SMS and WeChat (e.g., "Task T20250710002 failed to retry 3 times, anomaly type: file lock, please handle immediately"); and isolates the 100,000 fund transaction data records involved in the task to the "abnormal data pool" to avoid affecting subsequent task processing. For example, after a bank's end-of-day reconciliation task fails to retry 3 times, an emergency alarm is triggered and the associated batch payment tasks are suspended. In the healthcare field, the thresholds are set according to the urgency of the task: the retry threshold for emergency treatment data processing tasks is 2 times, and for physical examination report batch processing tasks it is 4 times. When the task of integrating the test results of an emergency patient fails twice consecutively due to "cross-department data interface timeout," the system: sends a pop-up alarm to the emergency department doctor's workstation and simultaneously calls the on-duty engineer; marks the patient's test data as "urgent abnormality" and stores it separately in a high-priority data partition. For example, if the task of processing real-time ECG data of an ICU patient fails twice after retrying, a hospital-wide alarm is triggered and a backup data channel is activated.

[0060] Specifically, in step S107, manual intervention achieves precise correction of abnormal data and task rescheduling through a visual interface. In the fintech field, the anomaly handling interface provides: detailed data display (e.g., presenting fields such as "transaction amount" and "merchant ID" for 100 abnormal transaction data in a table format); online editing functions (e.g., modifying "incorrect merchant classification codes"); and a reason labeling column (e.g., selecting reasons such as "system bug" or "data entry error"). After the operations and maintenance personnel correct a "transfer record with a negative amount," they can click the "re-execute" button, and the system will automatically move the task from the abnormal queue to the normal queue and prioritize its scheduling. For example, if an insurance company's policy rate calculation task is abnormal due to "incorrect age format for the insured ('35 years old' should be '35')," it can be recalculated within 5 minutes after manual correction. In the healthcare field, the management interface supports: medical data visualization (e.g., displaying "white blood cell count: 15.6 × 10⁻⁶" in an abnormal test report). 9 / L” and reference range); medical terminology standardization tools (e.g., correcting “high blood pressure” to “hypertension (ICD-10: I10)”); approval process (e.g., attending physician confirming the correction). After a doctor corrects a medical record that “mistakenly labeled ‘penicillin allergy’ as ‘no allergy’” through the interface, the doctor submits it to the system to re-execute the treatment plan recommendation task. For example, if the prescription review task for outpatients is abnormal due to “incorrect drug dosage unit ('5g' should be '5mg')”, the pharmacist corrects it and immediately re-executes the review and approves it.

[0061] Understandably, a comprehensive mechanism that monitors data processing task status, automatically recovers from abnormal tasks, triggers anomaly handling procedures, and facilitates manual intervention provides a more precise guarantee of the continuity and accuracy of data processing tasks compared to methods lacking task monitoring or relying solely on manual anomaly handling. This mechanism is particularly crucial in the fintech field because real-time monitoring of task status, such as transaction data processing and policy data calculation, timely handling of anomalies, and the introduction of manual intervention when necessary, help financial institutions ensure the integrity of business data processing and provide reliable guarantees for core businesses such as fund clearing and risk assessment. This task management framework based on comprehensive monitoring and anomaly handling demonstrates strong robustness across various types of financial data processing tasks. Whether it's high-frequency transaction data processing, large-scale policy accounting, or cross-institutional data synchronization, as long as financial data processing tasks are involved, this mechanism can enable timely detection and effective handling of anomalies, thereby ensuring the smooth operation of business processes. Unlike traditional anomaly handling that only passively responds after task failure, this method's real-time monitoring and automatic recovery features significantly improve the fault tolerance and processing efficiency of financial data processing tasks. After handling abnormal tasks, a crucial foundation is laid for the accurate application of subsequent financial business data and the scientific nature of decision-making. For example, in a bank's real-time payment system, this mechanism can promptly detect and handle processing anomalies in payment data, ensuring that payment tasks are resumed quickly and compliant payment results are generated. This enhances the service reliability and customer trust of financial institutions, effectively meeting the financial industry's demands for high availability and accuracy in data processing tasks. In the healthcare field, this end-to-end anomaly handling mechanism can monitor the status of tasks such as diagnostic data processing and medical insurance cost calculation in real time, promptly handle anomalies, and introduce manual intervention when necessary. This helps medical institutions ensure the integrity and accuracy of medical data processing, providing reliable data support for clinical diagnosis and medical insurance settlement. This task management framework based on end-to-end monitoring and anomaly handling is highly robust to various types of medical data processing tasks. Whether it's real-time vital sign data processing, laboratory report result calculation, or cross-regional medical insurance data synchronization, as long as medical data processing tasks are involved, this mechanism can enable timely detection and effective handling of anomalies, thereby ensuring the smooth operation of medical business processes. Unlike traditional anomaly handling that only passively responds after task failure, this method's real-time monitoring and automatic recovery characteristics significantly improve the fault tolerance and processing efficiency of medical data processing tasks. After handling abnormal tasks, this mechanism provides crucial support for the accurate application of subsequent medical data and the high-quality delivery of diagnostic and treatment services. For example, in a hospital's laboratory result transmission system, this mechanism can promptly detect and address transmission anomalies in a laboratory report, ensuring that test results are accurately delivered to the doctor's workstation within a short timeframe. This generates diagnostic and treatment information that meets clinical needs, improves the efficiency of medical institutions' diagnostic and treatment processes, enhances the patient experience, and effectively meets the healthcare sector's demands for high availability and accuracy in data processing tasks.

[0062] Step S120: Extract the basic data of the original business data and the corresponding correlation ratio parameters from the preset distributed database based on the preset big data platform.

[0063] It should be noted that the data extraction mechanism in step S120 is based on the distributed collaborative processing technology of a pre-set big data platform. Its core lies in accurately extracting key indicators from business data through threaded parallel operations and correlated queries. This adapts to scenarios such as premium calculation and reinsurance allocation in the fintech field, as well as scenarios such as medical expense settlement and medical insurance payment ratio calculation in the healthcare field. In the financial sector, "preserved original premium," "contractual ceded premium," and "contractual original premium" are key data for refined premium management and risk allocation. In the healthcare field, these concepts can be mapped to core indicators such as basic medical service costs, medical insurance reimbursement amounts, and total medical expenses, providing data support for expense settlement.

[0064] In some embodiments of the present invention, such as Figure 2 As shown, step S120 includes the following steps:

[0065] Step S121: Create a data processing thread using a preset program;

[0066] Step S122: Scan the business data table in batches from each shard node of the preset distributed database to obtain business data records;

[0067] Step S123: Extract the original premium for preservation as the basic data from the record field of each of the business data records;

[0068] Step S124: Query the association table of each of the business data records to obtain the contract ceded premium and the original contract premium corresponding to the original premium of the preservation as association ratio parameters.

[0069] Specifically, in step S121, the creation of data processing threads adopts a dynamic resource scheduling mechanism to adapt to the differentiated needs of the financial and medical fields. In the fintech field, thread pools are created based on the amount of policy data (e.g., one thread for every 500,000 policy records) using Java multithreading frameworks or Spark's TaskScheduler interface. For example, when processing annual premium settlement, 10 parallel threads are automatically created, each allocated 4GB of memory and 2 CPU cores to ensure efficient processing of massive amounts of policy data. Simultaneously, thread priority weights (e.g., a weight value of 1.5) are set for high-priority business (e.g., VIP customer policy maintenance) to prioritize the use of computing resources. In the healthcare field, thread creation is combined with the real-time requirements of medical data. Dedicated threads are created for different types of medical data using Python's concurrent.futures module or Dask's parallel computing interface: low-latency threads (response time ≤ 100ms) are created for real-time ECG monitoring data; and IO-intensive threads (configured with a 128MB pre-read cache) are created for batch extraction of historical medical record data. When the thread starts, a privacy protection plugin is loaded, which automatically enables the AES-256 encryption algorithm for sensitive data containing patient ID numbers and medical record content, in compliance with the data security requirements of the "Regulations on Medical Record Management of Medical Institutions".

[0070] Specifically, in step S122, the batch scan employs a sharded parallel strategy, optimizing the scanning rules based on the characteristics of the domain data. In the fintech field, a pre-defined distributed database (such as HBase or Greenplum) is used to shard business data tables (such as "Policy Maintenance Business Table" and "Contract Reinsurance Table") according to "Policy Effective Date + Region Code". Batch scan requests are initiated to each shard node (such as the RegionServer in East China) using SparkSQL's scan operator or Hive's mapreduce task. During the scan, business filtering rules are enabled, retaining only records with "Policy Status = Effective" and "Policy Type = Long-Term Life Insurance". The single-node scan rate can reach 2 million records per minute. For example, when scanning the 2024 East China Policy Maintenance Business Table, lapsed policies and short-term accident insurance records are filtered out, only valid data is obtained. In the healthcare field, distributed databases (such as CockroachDB and Amazon Aurora) store medical data tables (e.g., "outpatient expense tables" and "inpatient settlement tables") in shards based on "patient ID + year of visit." These shards are then scanned in batches using Flink's DataStream interface or Presto's distributed query engine. During scanning, medical data quality rules are applied to remove abnormal records such as "expense amount < 0" or "invalid treatment item code." Furthermore, medical terminology standardization tools (such as SNOMEDCT encoding mapping) are used to convert textual diagnoses like "common cold" and "upper respiratory tract infection" into standardized codes, ensuring data consistency. For example, when scanning the 2024 inpatient expense table of a top-tier hospital, 300 abnormal records caused by coding errors were automatically corrected.

[0071] Specifically, in step S123, basic data extraction is achieved through field mapping and validation mechanisms to ensure data accuracy. In the fintech field, values ​​are extracted from specified fields of business data records (such as the original premium field of the "Preservation Business Table," field ID = BF001). Regular expressions (such as ^\d+(\.\d{1,2})?$) are used to validate the data format, filtering out non-numeric values ​​and outliers exceeding two decimal places. Simultaneously, the extracted values ​​are verified against the insurance product rate table to ensure they are within a reasonable range (e.g., the original premium for critical illness insurance preservation must not be less than 500 yuan). For example, if the original premium for preservation extracted from a policy's preservation record is 3500.50 yuan, and the format and business rules are verified to be compliant, it is determined to be valid basic data.

[0072] In the healthcare field, "premium insurance premium" is mapped to "basic medical service fees." Values ​​are extracted from the fee field of business data records (such as the `clinic_item_cost` field in the "outpatient expense table"). Through medical insurance catalog verification rules, it is ensured that the extracted fees correspond to "Class A / Class B medical insurance items" (such as "complete blood count" and "cephalosporin antibiotics"), while excluding fees outside the medical insurance scope such as "cosmetic surgery." For example, if the basic fee for an "electrocardiogram" is 30 yuan extracted from a patient's outpatient record, and verification confirms it as a Class A medical insurance item, it is determined to be valid basic data.

[0073] Specifically, in step S124, the join query is implemented through primary key join and index optimization to ensure the accuracy of parameter extraction. In the fintech field, business data records are joined to the "contract reinsurance table" through the "policy number" (e.g., policy_no = P20240600123). Using MySQL's JOIN statement or MongoDB's lookup aggregation operation, the corresponding contract reinsurance premium (e.g., 500 yuan) and original contract premium (e.g., 2500 yuan) are queried. To accelerate the query, a B+ tree index or hash index is created in the "policy number" field, reducing the join query response time from seconds to milliseconds. For example, after a join query on the policy_no of a certain insurance policy, the reinsurance premium of 500 yuan and the original premium of 2500 yuan are obtained, which are used to calculate the reinsurance sharing ratio (500 / 2500 = 20%).

[0074] In the healthcare field, "contractual premiums" are mapped to "medical insurance reimbursement amounts," and "original contract premiums" are mapped to "total medical expenses." By linking "visit ID" to the "medical insurance settlement table," the corresponding medical_insurance_payment (e.g., 1400 yuan) and total_medical_cost (e.g., 2000 yuan) can be queried. Medical insurance policy filtering is enabled during the linked query, such as only querying records with "medical insurance status = normal participation" and "settlement year = 2024." The linked data is quickly located using the DRG grouping index (e.g., drg_group_code = E123). For example, after a patient's hospitalization record is linked using visit_id = V2024050088, the medical insurance reimbursement of 1400 yuan and the total medical expenses of 2000 yuan are obtained, which are used to calculate the medical insurance reimbursement ratio (1400 / 2000 = 70%).

[0075] Understandably, through the aforementioned distributed data extraction mechanism, compared to the traditional single-node serial processing method, the efficiency of premium data extraction in the financial sector is improved by 8-10 times, and the accuracy of medical expense data extraction in the medical sector is improved to 99.9%. In the fintech field, this mechanism supports core businesses such as automatic reinsurance and real-time premium calculation. For example, after applying it, an insurance company shortened its annual premium settlement time from 3 days to 8 hours. In the healthcare field, it assists in the DRG / DIP payment reform of medical insurance. A provincial medical insurance platform used this mechanism to achieve rapid extraction and proportion calculation of 120 million medical records across the province, improving settlement efficiency by 60%. This cross-domain adaptable extraction framework not only meets the high throughput requirements of financial data but also ensures the accuracy and security of medical data, providing a solid foundation for data-driven business decisions.

[0076] Step S130: Based on a preset proportional relationship, the basic data and the associated proportional parameters are calculated in parallel to generate data processing results.

[0077] It should be noted that the parallel computing mechanism in step S130 is a distributed collaborative computing technology based on a preset proportional relationship. Its core lies in achieving efficient fusion calculation of basic data and related proportional parameters through business data diversion and dynamic allocation of computing resources. In the fintech field, this can be used for scenarios such as premium sharing and risk reserve provisioning; in the healthcare field, it is suitable for scenarios such as calculating the medical insurance reimbursement ratio for medical expenses and cost sharing among multiple institutions. This mechanism ensures both the processing efficiency of large-scale data and the consistency of data through the aggregation of calculation results.

[0078] In some embodiments of the present invention, such as Figure 3 As shown, step S130 includes the following steps:

[0079] Step S131: For business data records that meet the preset diversion conditions, schedule them to the big data computing platform for distributed processing;

[0080] Step S132: In the big data computing platform, the proportional relationship is executed through a parallel computing framework to calculate and generate distributed computing results;

[0081] Step S133: Aggregate the distributed computing results with the local computing results to generate the data processing results.

[0082] Specifically, in step S131, the routing conditions are set in conjunction with the scale and complexity of the business data to achieve a reasonable allocation of computing tasks. In the fintech field, the preset routing conditions can be set to "the number of business data records in a single batch is ≥100,000" or "the associated ratio parameter involves 3 or more reinsurance institutions." When the policy maintenance business data meets the above conditions, the task is scheduled to the Spark cluster through the YARN resource manager. For example, if an insurance company has 500,000 monthly policy maintenance data records and involves the calculation of the ceding ratio of 10 reinsurance institutions, the system automatically triggers a distributed processing flow to shard the data to 8 computing nodes. In the healthcare field, the routing conditions can be set to "the number of daily medical records in a single institution is ≥50,000" or "involving cross-regional medical insurance settlement." When the hospital outpatient expense data meets the conditions, it is scheduled to the Flink cluster through the Kubernetes container orchestration platform. For example, if a provincial medical group has 150,000 daily outpatient data records and includes cross-provincial medical insurance patient settlement records, the system distributes the data to 12 computing nodes for processing to ensure settlement efficiency.

[0083] In some embodiments of the present invention, step S131 specifically includes: collecting system load indicators of the distributed database in real time; when the system load indicators exceed a first preset threshold, adjusting the flow distribution condition parameters to increase the task allocation ratio of the big data computing platform; when the system load indicators are lower than a second preset threshold, adjusting the flow distribution condition parameters in reverse to increase the task processing capacity of local computing resources.

[0084] It should be noted that the dynamic load balancing mechanism in step S131 is based on real-time system load perception and resource elastic scheduling strategies. It achieves optimal allocation of computing tasks between the big data platform and local resources by dynamically adjusting load balancing condition parameters. In the fintech field, this can cope with computing power fluctuations during peak premium settlement periods; in the healthcare field, it can adapt to load changes during medical insurance settlement windows, ensuring efficient utilization of computing resources.

[0085] Specifically, in step S131, a multi-dimensional monitoring scheme is adopted when collecting system load indicators of the distributed database in real time. In the fintech field, monitoring indicators include CPU utilization (e.g., threshold set to 70%), memory usage (e.g., threshold set to 80%), IOPS (e.g., threshold for input / output operations per second set to 10,000), and task queue length (e.g., threshold set to 5,000), with data collected once per second using monitoring tools such as Zabbix or Prometheus. For example, if an insurance company's distributed database CPU utilization spikes to 85% (exceeding the first preset threshold of 70%) during the peak premium settlement period at the beginning of the month, it will trigger an adjustment of the load distribution parameters. In the healthcare field, monitoring indicators cover the number of database connections (e.g., threshold set to 1,000), query response time (e.g., threshold set to 500ms), and storage throughput (e.g., threshold set to 500MB / s), with load changes visualized in real time using tools such as Grafana. For example, if a provincial medical insurance platform's database query response time reaches 800ms (exceeding the first preset threshold of 500ms) at the initial opening of the medical insurance settlement window, it will initiate an adjustment of the task allocation ratio.

[0086] When system load exceeds a first preset threshold, the task allocation parameters are adjusted to increase the task distribution ratio on the big data computing platform. In the financial scenario, the original allocation condition was "≥100,000 records per batch," which was adjusted to "≥50,000 records per batch," allowing more tasks to be scheduled to the Spark cluster. For example, 80,000 insurance policy records originally processed locally were moved to distributed processing on the big data platform, reducing local computing pressure and lowering database CPU utilization from 85% to 65%. In the healthcare scenario, the original allocation condition was "≥50,000 records per institution per day," which was adjusted to "≥30,000 records per institution per day," allowing more medical expense data to be processed by the Flink cluster. For example, 40,000 outpatient expense records from a hospital, originally computed locally, were moved to the big data platform, reducing database query response time from 800ms to 400ms.

[0087] When the system load index falls below the second preset threshold, the load balancing parameters are adjusted in reverse to increase the workload of local computing resources. In a financial scenario, if the database CPU utilization drops to 40% (below the second preset threshold of 50%), the load balancing condition is restored from "≥50,000 records" to "≥100,000 records," allowing local resources to process policy data in the range of 50,000 to 100,000 records. For example, 50,000 small-amount insurance policy records can be processed locally, reducing the resource consumption of the big data platform and improving resource utilization efficiency. In a medical scenario, if the database query response time drops to 300ms (below the second preset threshold of 400ms), the load balancing condition is restored from "≥30,000 records" to "≥50,000 records," allowing 30,000 to 50,000 medical records to be processed locally. For example, 40,000 outpatient records from a hospital can be reverted to local processing, avoiding idle big data platform resources and reducing computing costs.

[0088] Understandably, a dynamic task allocation mechanism based on system load metrics offers a more flexible approach to optimizing the distribution of computing tasks between the big data platform and local resources compared to task allocation methods with fixed allocation conditions. This mechanism is particularly crucial in the fintech sector, as accurately sensing load changes in the distributed database during peak premium settlement periods and dynamically adjusting the processing path of policy data helps financial institutions efficiently utilize computing resources, providing stable computing power support for monthly financial settlements, reinsurance allocation, and other business operations. This load-aware dynamic scheduling framework demonstrates strong robustness across various financial data processing scenarios. Whether it's large-scale premium calculations at the beginning of the month, routine sporadic policy maintenance, or sudden batch policy modifications, as long as computing task allocation is involved, real-time load monitoring can adjust the allocation strategy, thereby achieving efficient resource utilization. Unlike traditional fixed allocation methods that only allocate tasks based on data size, this method's real-time system load response significantly enhances the dynamic adaptability of financial computing resources. After the reasonable allocation of tasks, a crucial foundation is laid for the rapid processing of subsequent financial data and the smooth operation of business. For example, in the quarterly settlement of insurance companies, the task allocation ratio can be automatically adjusted based on changes in database load, ensuring that settlement work can still proceed efficiently under fluctuating computing power, generating financial data that meets regulatory requirements, improving the business continuity of financial institutions and the efficiency of cross-departmental computing power collaboration, and effectively meeting the financial industry's needs for flexible scheduling of computing resources and efficient processing of large-scale business data. In the healthcare field, this dynamic load sharing mechanism can accurately perceive changes in the load of the distributed database at the medical insurance settlement window, dynamically adjust the processing path of medical expense data, help medical insurance departments and medical institutions complete expense settlements efficiently, and provide stable computing power guarantees for the smooth operation of medical services. This load-aware dynamic scheduling framework is highly robust to various types of medical data processing scenarios. Whether it is the massive expense calculation when the medical insurance settlement window is open, the real-time settlement of daily outpatient expenses, or the processing of cross-regional medical expenses, as long as medical data computing task allocation is involved, the load sharing strategy can be adjusted through real-time load monitoring, thereby achieving efficient utilization of resources. Unlike the traditional fixed load sharing method that only allocates tasks according to data scale, this method's real-time system load response characteristics significantly improve the dynamic adaptability of medical computing resources. After the tasks are properly allocated, it provides crucial support for the rapid settlement of subsequent medical expenses and the accurate supervision of medical insurance funds. For example, in the daily operation of a provincial medical insurance platform, the task allocation ratio can be automatically adjusted according to changes in database load to ensure that expense settlement can still be completed quickly during peak periods of medical visits, generating settlement results that comply with medical insurance policies. This improves the timeliness of medical insurance services and the collaborative processing capabilities of cross-regional medical data, effectively meeting the needs of the healthcare sector for flexible scheduling of computing resources and efficient processing of large-scale medical data.

[0089] Specifically, in step S132, the execution of the proportional relationship formula adopts a framework adaptation strategy, combining domain business rules to ensure calculation accuracy. In the fintech field, for premium sharing scenarios, the preset proportional relationship formulas can be "Reinsurance Ratio = Contractual Ceased Premium ÷ Original Contractual Premium × 100%" and "Shared Premium = Original Premium × Reinsurance Ratio," which are implemented in parallel through Spark's RDD operators or DataFrame API. For example, in a Spark cluster with 500,000 data entries, each computing node processes 60,000-70,000 records in parallel, calculating the shared premium for each record according to the reinsurance ratio formula, with a single node processing 8,000 records per second. In the healthcare field, for medical insurance reimbursement calculations, the proportional relationship formulas can be "Medical Insurance Reimbursement Amount = Basic Cost of Treatment Item × Medical Insurance Payment Ratio" and "Personal Out-of-Pocket Amount = Basic Cost of Treatment Item - Medical Insurance Reimbursement Amount," which are calculated through Flink's DataStream or Table API. For example, 150,000 outpatient expense records are stored in a Flink cluster and calculated in parallel by patient medical insurance type (employee medical insurance, resident medical insurance). Employee medical insurance is calculated at a 70% reimbursement rate and resident medical insurance is calculated at a 50% reimbursement rate. During the calculation process, the project restrictions in the medical insurance catalog (such as the annual reimbursement limit for certain examination items) are automatically associated.

[0090] Specifically, in step S133, the aggregation of calculation results adopts a multi-source data fusion strategy to ensure the integrity and accuracy of the results. In the fintech field, distributed computing results (such as the premium allocation of each reinsurance agency output by a Spark cluster) and local computing results (such as the calculation results of small policy fees processed in a small database) are associated through the "policy number" primary key. Apache Kafka is used as a message middleware to achieve real-time data aggregation, and then the aggregation operation is performed through Hive or ClickHouse. For example, 500,000 distributed premium allocation records and 20,000 local small premium records are grouped and summarized by "reinsurance agency + policy type" to generate a total premium allocation report for each agency. A data verification mechanism is enabled during the aggregation process to ensure that the total amount is consistent with the original premium amount. In the healthcare field, distributed computing results (such as the medical insurance reimbursement amount of each department output by a Flink cluster) and local computing results (such as the out-of-pocket expenses processed by the hospital's local system) are associated through the "patient ID". Intermediate results are temporarily stored using Redis caching, and then aggregation is performed through Python's Pandas library. For example, 150,000 medical insurance reimbursement amounts calculated by distributed computing and 30,000 out-of-pocket item amounts calculated locally are aggregated by "patient ID + date of visit" to generate a total patient expense list. At the same time, the logical relationship between medical insurance reimbursement amounts, basic medical service fees, and medical insurance payment ratios is verified to avoid calculation errors.

[0091] Understandably, parallel computing mechanisms based on preset proportional relationships can more efficiently handle the fusion calculations of large-scale business data and complex proportional parameters compared to traditional single-local computing or fully distributed computing. This mechanism is particularly crucial in the fintech field because accurately calculating data such as premium allocation and risk reserves helps financial institutions quickly complete monthly and quarterly financial settlements, providing precise data support for business decisions and regulatory compliance reports. This business-based parallel computing framework demonstrates strong adaptability to various types of financial data. Whether it's premium allocation for massive insurance policies or reinsurance ratio calculations involving multiple institutions, as long as the preset proportional relationship rules are met, accurate results can be generated through split processing and parallel computation, thereby achieving efficient financial accounting. Unlike traditional computing methods that only focus on the processing efficiency of a single data scale, this method's combination of distributed and local computing significantly improves the flexibility and overall efficiency of financial data computing. After generating the data processing results, it forms an important foundation for subsequent financial statement preparation and the application of risk assessment models. For example, in the annual reinsurance settlement of insurance companies, the generated processing results can automatically aggregate the allocated amounts of each reinsurance institution, generating financial reports that meet regulatory requirements. This improves the automation level of financial processing in financial institutions and the consistency of cross-departmental data collaboration, effectively meeting the financial industry's needs for accurate calculation and efficient settlement of large-scale business data. In the healthcare field, this parallel computing mechanism can accurately calculate data such as the medical insurance reimbursement amount for medical expenses and the cost-sharing ratio among multiple medical institutions, helping medical insurance departments and medical institutions to efficiently complete cost settlements and providing a scientific basis for the rational allocation of medical resources and the supervision of medical insurance funds. This parallel computing framework based on proportional relationships is highly robust to complex medical expense data. Whether it is the calculation of massive outpatient expenses at a single institution or cross-regional medical insurance settlement, as long as it conforms to the calculation rules of medical expenses, accurate cost settlement results can be generated through diversion processing and parallel computing, thereby achieving efficient medical expense management. Unlike the traditional medical expense calculation method that relies on manual calculation or single-system processing, the aggregation characteristics of this method significantly improve the accuracy and timeliness of medical data calculation. Once the data processing results are generated, they provide crucial support for the rational use of medical insurance funds and cost control in medical institutions. For example, in the cost settlement process of the provincial medical insurance platform, the generated processing results can automatically calculate the medical insurance payment amount for each hospital, generate settlement reports that comply with medical insurance policies, improve the efficiency of medical insurance settlement and the collaborative processing capability of cross-regional medical insurance data, and effectively meet the needs of the medical and health field for accurate calculation and efficient cost settlement of large-scale medical data.

[0092] Step S140: Write the data processing result back to the specified data table of the preset distributed database.

[0093] It should be noted that the data write-back mechanism in step S140 is a closed-loop processing step based on distributed transactions and data consistency verification. Its core is to securely and accurately store the calculated processing results into a designated data table in a pre-defined distributed database, achieving complete data processing integrity. In the fintech field, it can be used to write back premium calculation results and risk assessment data to the business database; in the healthcare field, it is suitable for writing back medical expense settlement results and medical insurance reimbursement ratio calculation results to the medical data center, providing data support for subsequent business queries and analysis.

[0094] In some embodiments of the present invention, step S140 is specifically operated as follows:

[0095] In the fintech field, the write-back process employs a "partition location + transaction protection" strategy. First, based on the business type corresponding to the data processing results (e.g., life insurance premium calculation, auto insurance claims calculation), the data is located in the corresponding designated data table in the distributed database (e.g., "Life Insurance Premium Result Table" and "Auto Insurance Claims Result Table"). These tables are stored in partitions based on "policy year + region code." During write-back, distributed transactions (e.g., based on the 2PC protocol) are enabled to ensure consistency between the calculation results and the associated original business data. For example, when writing back the premium allocation result of a policy, the logical relationship between the result and the original premium amount and allocation ratio is simultaneously verified to avoid data conflicts. Simultaneously, an "incremental write-back" mode is adopted, updating only the changed fields (e.g., "allocation amount" and "update timestamp") to reduce data transmission volume. For instance, when an insurance company writes back 100,000 auto insurance claims calculation results, it accurately locates the corresponding partition using the policy number, and only updates the "claim amount" and "payment status" fields for each result, reducing write-back time compared to a full write.

[0096] In the healthcare field, the write-back process employs a "tiered authorization + version control" strategy in conjunction with medical data standards. Based on the type of processing result (e.g., outpatient expense settlement, inpatient medical insurance reimbursement), the system locates a specific data table (e.g., "Outpatient Expense Settlement Result Table" or "Inpatient Medical Insurance Reimbursement Result Table"), which is partitioned by "Patient ID + Visit Date". Before writing back, the system verifies the operator's permissions (e.g., only medical insurance settlement specialists can write to the medical insurance reimbursement result table). During the write-back, a version control mechanism is enabled, adding a version number (e.g., "V1.0" or "V2.0") to each processing result, and recording the timestamp and operator information for each write-back. Simultaneously, data format validation is performed to ensure that the written-back settlement results conform to medical data standards (e.g., amounts are rounded to two decimal places, and medical insurance catalog codes match). For example, when a hospital writes back 5,000 outpatient expense settlement results, the system automatically checks whether the "medical insurance reimbursement amount" in each result is within the "total cost" range. Records that do not conform trigger an alarm and pause the write-back process, resuming execution only after manual verification. After the write-back is complete, the system performs a double check: first, it checks whether the amount of data written back matches the total number of processed records to avoid data omissions; second, it randomly selects a portion of records and compares the write-back results with the processed results temporarily stored in memory to ensure that the data has not been tampered with. In the financial sector, after successful verification, a write-back success report is generated, including information such as the write-back time, the number of records involved, and no abnormal records, for the auditing department to review. In the medical field, after successful verification, the results are synchronized to the electronic medical record system, and doctors can query the latest cost settlement results through the patient ID, providing a reference for treatment decisions.

[0097] Understandably, a data write-back mechanism employing precise positioning, transaction safeguards, and multi-factor verification can more accurately ensure the accuracy and integrity of data processing results in a distributed database compared to simple batch writing or manual entry. This mechanism is particularly crucial in the fintech field, as accurately writing premium calculation results and risk assessment data back to designated data tables helps financial institutions build complete business data chains, providing reliable data source support for financial audits and business reviews. This write-back framework based on distributed transactions and consistency verification demonstrates strong robustness to various types of financial data processing results. Whether it's the calculation results of high-frequency transactions, the accounting data of large-scale insurance policies, or cross-institutional settlement information, as long as it pertains to financial business processing results, this mechanism can achieve secure and accurate storage, thereby ensuring the traceability and consistency of financial data. Unlike traditional write-back methods that only focus on the success of data storage, this method's transaction constraints and multi-factor verification significantly improve the reliability and security of financial data write-back. After completing the data write-back, it forms an important foundation for subsequent data analysis and decision-making in financial business. For example, in a bank's annual financial settlement, the standardized data written back can be used to quickly summarize the calculation results of various business lines, generate financial statements that meet regulatory requirements, improve the data governance level and cross-departmental data collaboration efficiency of financial institutions, and effectively meet the financial industry's needs for accurate storage and in-depth application of data processing results. In the healthcare field, this data write-back mechanism can accurately write back the results of medical expense settlements and medical insurance reimbursement ratio calculations to designated data tables, helping medical institutions and medical insurance departments build a complete medical data chain and providing reliable data source support for clinical diagnosis and treatment review and medical insurance fund supervision. This write-back framework based on hierarchical authorization and version control is highly robust to various types of medical data processing results. Whether it is real-time updated outpatient expense settlement results, batch-processed inpatient medical insurance reimbursement data, or cross-regional medical treatment settlement information, as long as it belongs to the medical business processing results, it can be securely and accurately stored through this mechanism, thereby ensuring the traceability and consistency of medical data. Unlike traditional write-back methods that only focus on whether the data is stored successfully, this method's access control and version tracking features significantly improve the standardization and security of medical data write-back. After the data is written back, it provides crucial support for subsequent medical data analysis and medical service optimization. For example, in hospital medical quality assessment, the standardized data written back can be used to quickly summarize the cost settlement results of each department, generate assessment reports that comply with medical standards, improve the data governance level of medical institutions and the efficiency of cross-departmental data collaboration, and effectively meet the needs of the medical and health field for accurate storage and in-depth application of data processing results.

[0098] In summary, the solution implemented in the embodiments of the present invention can solve the problems of low processing efficiency and difficulty in guaranteeing accuracy in traditional data processing methods used in the prior art.

[0099] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0100] In one embodiment, a data processing apparatus is provided, which corresponds one-to-one with the data processing methods described in the above embodiments. For example... Figure 4 As shown, the data processing device includes an acquisition module 410, an extraction module 420, a generation module 430, and a write-back module 440. Detailed descriptions of each functional module are as follows:

[0101] The acquisition module 410 is used to acquire raw business data and store the raw business data in a preset distributed database.

[0102] Extraction module 420 is used to extract basic data and corresponding correlation ratio parameters of the original business data from the preset distributed database based on the preset big data platform;

[0103] The generation module 430 is used to generate data processing results by parallel calculation of the basic data and the associated proportional parameters based on a preset proportional relationship.

[0104] The write-back module 440 writes the data processing results back to a specified data table in the preset distributed database.

[0105] In one embodiment, the extraction module 420 is specifically used for:

[0106] A data processing thread is created using a pre-defined program;

[0107] The business data records are obtained by batch scanning the business data tables from each shard node of the preset distributed database.

[0108] Extract the original premium for insurance preservation from the record fields of each of the aforementioned business data records as the basic data;

[0109] The contract ceded premium and contract original premium corresponding to the original premium of the preservation are retrieved from the association table of each of the aforementioned business data records as association ratio parameters.

[0110] In one embodiment, the generation module 430 is specifically used for:

[0111] For business data records that meet the preset diversion conditions, they are scheduled to the big data computing platform for distributed processing;

[0112] In the big data computing platform, the proportional relationship is executed through a parallel computing framework to calculate and generate distributed computing results.

[0113] The data processing result is generated by aggregating the results of distributed computing with the results of local computing.

[0114] In one embodiment, the generation module 430 is further configured to:

[0115] Real-time collection of system load metrics for the distributed database;

[0116] When the system load index exceeds the first preset threshold, the traffic splitting condition parameters are adjusted to increase the task allocation ratio of the big data computing platform.

[0117] When the system load index is lower than the second preset threshold, the diversion condition parameters are adjusted in reverse to increase the task processing capacity of local computing resources.

[0118] In one embodiment, the acquisition module 410 is further configured to:

[0119] Extract the original business data from the business source system;

[0120] The extracted raw business data is subjected to standardized data transformation processing to generate structured data;

[0121] After verifying the integrity and consistency of the structured data, the distributed storage process is triggered.

[0122] In one embodiment, the acquisition module 410 is further configured to:

[0123] Structured data is segmented based on business time characteristics;

[0124] Establish an independent transaction log recording mechanism for each data shard;

[0125] A distributed transaction protocol is used to verify cross-shard data consistency.

[0126] In one embodiment, the acquisition module 410 is further configured to:

[0127] Monitor the execution status of data processing tasks;

[0128] Automatic recovery processing is performed on abnormal tasks, wherein the automatic recovery processing includes marking the abnormal state and adding it to the task retry queue;

[0129] When the number of retries reaches a preset threshold, the exception handling process is triggered.

[0130] After manually intervening in the abnormal data through the aforementioned exception handling process management interface, the calculation task is re-executed.

[0131] This invention provides a solution offered by a data processing device. By centrally storing raw business data through a pre-set distributed database, it significantly improves the convenience and completeness of data acquisition compared to the traditional model of scattered storage across multiple independent systems. The device extracts and performs parallel computations on the raw business data, effectively integrating basic data and related proportional parameters to rapidly generate processing results, forming a high-quality, standardized business dataset. This avoids decision-making biases caused by data dispersion and computational lag, laying a solid foundation for subsequent accurate analysis. Leveraging a pre-set big data platform and proportional relationship formulas to deeply mine the value of business data, it can accurately extract key information and correlation patterns from the data. Compared to traditional simple analysis methods based on single data, it can more comprehensively and accurately support business decisions, and the generated data processing results are more aligned with actual business needs, helping industry players to anticipate business dynamics. Combining the data processing results with pre-set business rules enables automated and intelligent business management. This method breaks through the limitations of traditional data processing applications in the fintech and healthcare fields. In the fintech field, it helps financial institutions optimize risk control and improve operational efficiency, while in the healthcare field, it provides medical institutions with precise diagnosis and treatment support and optimizes resource allocation. It achieves a dual improvement in the quality and efficiency of data processing in cross-domain applications, providing new impetus for industry development.

[0132] In summary, the solution implemented by the above data processing method can solve the problems of low processing efficiency and difficulty in guaranteeing accuracy in traditional data processing methods used in the prior art.

[0133] Specific limitations regarding the data processing device can be found in the limitations regarding the data processing method described above, and will not be repeated here. Each module in the aforementioned data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0134] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements a recommended method for an optimal strategy, representing the functions or steps on the server side.

[0135] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements a recommended method for an optimal strategy, representing client-side functions or steps.

[0136] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0137] Obtain the original business data and store the original business data in a preset distributed database;

[0138] Based on a pre-defined big data platform, the basic data and corresponding correlation ratio parameters of the original business data are extracted from the pre-defined distributed database.

[0139] Based on a preset proportional relationship, the basic data and the associated proportional parameters are calculated in parallel to generate data processing results;

[0140] The data processing results are written back to a designated data table in the preset distributed database.

[0141] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0142] Obtain the original business data and store the original business data in a preset distributed database;

[0143] Based on a pre-defined big data platform, the basic data and corresponding correlation ratio parameters of the original business data are extracted from the pre-defined distributed database.

[0144] Based on a preset proportional relationship, the basic data and the associated proportional parameters are calculated in parallel to generate data processing results;

[0145] The data processing results are written back to a designated data table in the preset distributed database.

[0146] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0147] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0148] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0149] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A data processing method, characterized in that, include: Obtain the original business data and store the original business data in a preset distributed database; Based on a pre-defined big data platform, the basic data and corresponding correlation ratio parameters of the original business data are extracted from the pre-defined distributed database. Based on a preset proportional relationship, the basic data and the associated proportional parameters are calculated in parallel to generate data processing results; The data processing results are written back to a designated data table in the preset distributed database.

2. The data processing method according to claim 1, characterized in that, The basic data and corresponding correlation ratio parameters for extracting the original business data from the preset distributed database based on the preset big data platform include: A data processing thread is created using a pre-defined program; The business data records are obtained by batch scanning the business data tables from each shard node of the preset distributed database. Extract the original premium for insurance preservation from the record fields of each of the aforementioned business data records as the basic data; The contract ceded premium and contract original premium corresponding to the original premium of the preservation are retrieved from the association table of each of the aforementioned business data records as association ratio parameters.

3. The data processing method according to claim 1, characterized in that, The process of generating data processing results by parallel calculation of the basic data and the associated proportional parameters based on a preset proportional relationship includes: For business data records that meet the preset diversion conditions, they are scheduled to the big data computing platform for distributed processing; In the big data computing platform, the proportional relationship is executed through a parallel computing framework to calculate and generate distributed computing results. The data processing result is generated by aggregating the results of distributed computing with the results of local computing.

4. The data processing method according to claim 3, characterized in that, The process of scheduling business data records that meet the preset diversion conditions to the big data computing platform for distributed processing includes: Real-time collection of system load metrics for the distributed database; When the system load index exceeds the first preset threshold, the traffic splitting condition parameters are adjusted to increase the task allocation ratio of the big data computing platform. When the system load index is lower than the second preset threshold, the diversion condition parameters are adjusted in reverse to increase the task processing capacity of local computing resources.

5. The data processing method according to claim 1, characterized in that, Before acquiring the original business data and storing the original business data in a preset distributed database, the method further includes: Extract the original business data from the business source system; The extracted raw business data is subjected to standardized data transformation processing to generate structured data; After verifying the integrity and consistency of the structured data, the distributed storage process is triggered.

6. The data processing method according to claim 5, characterized in that, After verifying the integrity and consistency of the data, a distributed storage process is triggered, including: Structured data is segmented based on business time characteristics; Establish an independent transaction log recording mechanism for each data shard; A distributed transaction protocol is used to verify cross-shard data consistency.

7. The data processing method according to claim 5, characterized in that, After verifying the integrity and consistency of the structured data and triggering the distributed storage process, the process further includes: Monitor the execution status of data processing tasks; Automatic recovery processing is performed on abnormal tasks, wherein the automatic recovery processing includes marking the abnormal state and adding it to the task retry queue; When the number of retries reaches a preset threshold, the exception handling process is triggered. After manually intervening in the abnormal data through the aforementioned exception handling process management interface, the calculation task is re-executed.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire raw business data and store the raw business data in a preset distributed database. The extraction module is used to extract the basic data and corresponding correlation ratio parameters of the original business data from the preset distributed database based on the preset big data platform; The generation module is used to generate data processing results by parallel calculation of the basic data and the associated proportional parameters based on a preset proportional relationship. The write-back module writes the data processing results back to a specified data table in the preset distributed database.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the data processing method as described in any one of claims 1 to 7.

Images