Insurance data calculation method and device based on distributed engine, equipment and medium

Abstract

The invention relates to the technical field of cloud computing, and discloses an insurance data computing method, device and equipment based on a distributed engine and a medium, the method comprises the following steps: capturing an insurance operation log and a dimension logic table, associating the insurance operation log with the dimension logic table in real time, and processing to obtain insurance accumulated data of a current period; obtaining offline data and calculating insurance accumulated data of a previous period to obtain an offline data calculation result; pushing the offline data calculation result to an offline data table; and taking the off-line data table as a main table, and left-associating the off-line data table to the insurance accumulation data of the current period to obtain insurance fusion data. In the invention, aiming at the problem of poor stability and expansibility of a traditional quasi-real-time computing architecture, based on the distributed engine and the relational database, the insurance accumulation data of the current period and the insurance accumulation data of the previous period are calculated and obtained, and then the calculation results of the two are combined to obtain the insurance fusion data, so that the stability of the system is improved; and meanwhile, the expansion requirement of real-time calculation is supported.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to an insurance data calculation method, device, equipment and medium based on a distributed engine. Background Art

[0002] In the insurance industry, insurance institutions often launch more cost-effective products in order to cope with market competition. Such sales activities usually have the following characteristics: on the one hand, the premium amount of the products sold is high, which may be accompanied by the launch of new products; on the other hand, insurance institutions will centrally issue orders in the system after midnight. In this business scenario, it is required to be able to count the sales performance of institutions at all levels in real time and estimate the value, so as to follow up the business progress in real time. To this end, the real-time computing engine must be able to support fast recovery and be able to carry more computing needs during peak nodes.

[0003] Most existing technologies use a quasi-real-time computing architecture to mount tasks on the Linkdo platform. However, as streaming computing tasks continue to run, the amount of logs is also increasing, making task monitoring complicated and difficult. At the same time, the calculation results are written into traditional databases, but traditional databases are bound to the data interfaces of multiple application systems, which are highly correlated and not conducive to subsequent development and maintenance. This design has exposed the following deficiencies in long-term operation: First, as the amount of logs increases, the complexity of system monitoring and maintenance increases significantly, affecting the operational stability of tasks; Second, as business needs continue to change, the existing architecture is difficult to quickly respond to the expansion needs of real-time computing, especially when dealing with peak traffic.

[0004] Based on the above shortcomings, an improved real-time computing architecture is urgently needed to improve the stability and scalability of the system. Summary of the invention

[0005] The present invention provides an insurance data calculation method, device, equipment and medium based on a distributed engine to solve the technical problems of poor stability and scalability of traditional quasi-real-time computing architecture.

[0006] In a first aspect, a method for calculating insurance data based on a distributed engine is provided, comprising:

[0007] Capturing the insurance operation log of the relational database using structured data replication software, and writing the insurance operation log into the topic of the distributed management platform;

[0008] Importing the dimension logic table calculated in advance in the data warehouse tool into the relational database;

[0009] Using a distributed engine to obtain the insurance operation log from the subject of the distributed management platform, and perform real-time association with the dimension logic table to obtain a real-time data association result;

[0010] Processing the real-time data association result to obtain a real-time data calculation result, and writing the real-time data calculation result into a real-time data table of the relational database to obtain the current period insurance accumulation data;

[0011] Obtain offline data from the data operation layer in the data warehouse tool, and use the data warehouse tool to calculate the insurance accumulation data of the previous period to obtain offline data calculation results;

[0012] Pushing the offline data calculation result to the offline data table of the relational database through a data transfer tool;

[0013] The offline data table is used as the main table, and is left-linked to the current period insurance accumulation data to obtain insurance fusion data of the real-time data table and the offline data table;

[0014] A data view is created based on the relational database, and the data view is bound to a preset API service to provide the insurance fusion data to external applications.

[0015] In a second aspect, an insurance data computing device based on a distributed engine is provided, comprising:

[0016] A log acquisition module, used to capture the insurance operation log of the relational database using structured data replication software, and write the insurance operation log into the topic of the distributed management platform;

[0017] A log acquisition module, used for importing the dimension logic table pre-calculated in the data warehouse tool into the relational database;

[0018] A log acquisition module, used to acquire the insurance operation log from the subject of the distributed management platform by using a distributed engine, and associate it with the dimension logic table in real time to obtain a real-time data association result;

[0019] A real-time calculation module, used for performing data processing on the real-time data association result to obtain a real-time data calculation result, and writing the real-time data calculation result into the real-time data table of the relational database to obtain the current period insurance accumulation data;

[0020] An offline calculation module, used to obtain offline data from the data operation layer in the data warehouse tool, and use the data warehouse tool to calculate the insurance accumulation data of the previous period to obtain offline data calculation results;

[0021] A data migration module, used to push the offline data calculation results to the offline data table of the relational database through a data transfer tool;

[0022] A table association module is used to take the offline data table as the main table and left-associate it to the current period insurance accumulation data to obtain insurance fusion data of the real-time data table and the offline data table;

[0023] A data output module is used to create a data view based on the relational database, and bind the data view to a preset API service to provide the insurance fusion data to an external application.

[0024] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned insurance data calculation method based on a distributed engine when executing the computer program.

[0025] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned insurance data calculation method based on a distributed engine are implemented.

[0026] In the scheme implemented by the above-mentioned insurance data calculation method, device, equipment and medium based on the distributed engine, the insurance operation log of the relational database is captured by structured data replication software, and the insurance operation log is written into the subject of the distributed management platform; the dimension logic table calculated in advance in the data warehouse tool is imported into the relational database; the insurance operation log is obtained from the subject of the distributed management platform by using the distributed engine, and is associated with the dimension logic table in real time to obtain a real-time data association result; the real-time data association result is processed to obtain a real-time data calculation result, and the real-time data calculation result is written into the relational database. The real-time data table of the database is used to obtain the insurance cumulative data of the current period; offline data is obtained from the data operation layer in the data warehouse tool, and the data warehouse tool is used to calculate the insurance cumulative data of the previous period to obtain the offline data calculation result; the offline data calculation result is pushed to the offline data table of the relational database through the data transfer tool; the offline data table is used as the main table, and left-linked to the insurance cumulative data of the current period, to obtain the insurance fusion data of the real-time data table and the offline data table; a data view is created based on the relational database, and the data view is bound to the preset API service to provide the insurance fusion data to external applications. In the present invention, in view of the poor stability and scalability of the traditional quasi-real-time computing architecture, based on a distributed engine and a relational database, the insurance cumulative data of the current period and the insurance cumulative data of the previous period are calculated, and then the calculation results of the two are merged to obtain the insurance fusion data, which improves the stability of the system and supports the expansion requirements of real-time computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0028] Figure 1 It is a flow chart of a method for calculating insurance data based on a distributed engine in one embodiment of the present invention;

[0029] Figure 2 yes Figure 1 A schematic flow chart of a specific implementation of step S10;

[0030] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S20;

[0031] Figure 4 yes Figure 1A schematic flow chart of a specific implementation of step S40;

[0032] Figure 5 yes Figure 1 Another specific implementation flow diagram of step S40;

[0033] Figure 6 yes Figure 1 A schematic flow chart of a specific implementation of step S50;

[0034] Figure 7 yes Figure 1 A schematic flow chart of a specific implementation of step S70;

[0035] Figure 8 It is a structural schematic diagram of an insurance data calculation device based on a distributed engine in one embodiment of the present invention;

[0036] Fig. 9 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0037] Fig.10 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] See also Figure 1 As shown, Figure 1 A flow chart of a method for calculating insurance data based on a distributed engine provided in an embodiment of the present invention includes the following steps:

[0040] S10, using structured data replication software to capture the insurance operation log of the relational database, and writing the insurance operation log into the topic of the distributed management platform;

[0041] S20, importing the dimension logic table calculated in advance in the data warehouse tool into the relational database;

[0042] S30, using a distributed engine to obtain the insurance operation log from the subject of the distributed management platform, and perform real-time association with the dimension logic table to obtain a real-time data association result;

[0043] S40, performing data processing on the real-time data association result to obtain a real-time data calculation result, and writing the real-time data calculation result into the real-time data table of the relational database to obtain the current period insurance accumulation data;

[0044] S50, acquiring offline data from the data operation layer in the data warehouse tool, and using the data warehouse tool to calculate the insurance accumulation data of the previous period to obtain offline data calculation results;

[0045] S60, pushing the offline data calculation result to the offline data table of the relational database through a data transfer tool;

[0046] S70, taking the offline data table as the main table, and left-linking it to the current period insurance accumulated data, to obtain insurance fusion data of the real-time data table and the offline data table;

[0047] S80: Create a data view based on the relational database, and bind the data view to a preset API service to provide the insurance fusion data to external applications.

[0048] In step S10, first, the operation log of the insurance business is captured from the relational database using structured data replication software. The operation log records operation information such as adding, updating, and deleting, and writes it into the subject of the distributed management platform in the form of structured data. The capture of the operation log ensures the real-time flow of insurance business data, laying the foundation for subsequent real-time data calculation.

[0049] Combination Figure 2 As shown, the method of capturing the insurance operation log of the relational database using structured data replication software includes the following specific steps:

[0050] S11. Using the structured data replication software, identify data operations in the relational database.

[0051] In step S11, all data operations in the relational database can be monitored by the structured data replication software. The structured data replication software can detect and identify operations such as insert, update, and delete in the relational database in real time.

[0052] S12: extracting insert operations, update operations, and delete operations from the data operations, and generating corresponding log entries according to the operation types.

[0053] Among the captured database operations, the system will automatically identify the operation type and extract the insert operation, update operation, and delete operation. Each operation type corresponds to different data changes, and the system generates corresponding log entries based on the operation type. For example, the log entry for the insert operation will record the newly added data items, the update operation will record the data difference before and after the modification, and the delete operation will record the removed data items. These log entries are accurate records of changes in insurance business data.

[0054] S13. Convert the format of the log entry and store it in a log storage file according to a preset structured format to obtain the insurance operation log.

[0055] In order to ensure the readability and standardization of log data, the system will convert the format of log entries after they are generated. Specifically, the system converts various types of operation logs into a unified, structured format to ensure that key information in the log entries (such as operation type, operation time, operation content, etc.) can be correctly parsed and used by other system modules. After completing the format conversion, the system stores these log entries in a log storage file. The log storage file is stored in a preset structured format and serves as the basic data source for subsequent data processing.

[0056] In step S20, the dimension table of the business is pre-calculated offline using a data warehouse tool to obtain a dimension logic table. The dimension logic table contains various dimension information related to the insurance business, such as product attributes, institutional information, customer classification, etc. The calculated dimension logic table will be imported into the relational database for subsequent real-time data association calculation.

[0057] Combination Figure 3 As shown, the step of importing the dimension logic table pre-calculated in the data warehouse tool into the relational database includes:

[0058] S21. Design the hierarchical structure of the dimension logic table according to preset business requirements to obtain a fact table and a dimension table.

[0059] In step S21, according to business requirements, the hierarchical structure of the dimension logic table is first designed. In this process, data is divided into two categories according to business logic: one is the fact table used to record business events, and the other is the dimension table that describes the business dimension. The fact table usually includes core data directly related to business operations such as insurance policies and customers, while the dimension table provides attribute information describing the data in the fact table, such as product type, organization information, etc.

[0060] S22. Use a multidimensional data model to logically associate the fact table with the dimension table to obtain the dimension logic table.

[0061] In step S22, the fact table and the dimension table are logically associated using a multidimensional data model to generate a complete dimension logic table. In the multidimensional data model, the fact table is associated through key fields (such as product ID, organization ID, etc.) in the dimension table, so that each fact information can be aggregated and analyzed based on its corresponding dimension information. In this way, business analysis for different dimensions can be achieved to meet the diversified needs of insurance business.

[0062] S23. Based on the parallel computing framework of the data warehouse tool, a directed acyclic graph scheduling mechanism is used to define the execution order and dependency relationship of each computing task in the dimension logic table.

[0063] In step S23, based on the parallel computing framework of the data warehouse tool, a directed acyclic graph (DAG) scheduling mechanism is used to define the execution order and dependency of each computing task in the dimension logic table. In DAG scheduling, the system automatically arranges the calculation order according to the dependency of the task to ensure that each task can be executed in parallel in the established order to avoid calculation deadlock or dependency conflict. This scheduling mechanism not only improves the efficiency of data calculation, but also ensures the correctness of data processing.

[0064] S24: After the dimension logic table is calculated, different versions of the dimension logic table are partitioned and stored in the relational database.

[0065] After the dimension logic table is calculated, the system divides it into different versions for partition storage and imports it into the relational database. Each version of the dimension logic table corresponds to the calculation results in different time periods or different business scenarios. Through partition storage, the system can effectively manage data in different time periods and business dimensions, which not only ensures the flexibility of data but also improves the efficiency of queries. In addition, partition storage can reduce the occupation of database resources and improve the scalability of the system.

[0066] In step S30, the real-time captured insurance operation logs are obtained from the topic of the distributed management platform through a distributed engine (such as Flink, etc.). The distributed engine associates these insurance operation logs with the previously imported dimension logic table, performs real-time association calculations, and obtains real-time data association results.

[0067] In step S40, the obtained real-time data association results are further processed to finally obtain the real-time data calculation results of the insurance business. The processed real-time data calculation results are written into the real-time data table of the relational database for subsequent query and statistical use. At the same time, these data reflect the insurance accumulation data in the current period and can provide data support for real-time business decision-making.

[0068] Combination Figure 4As shown, the real-time data association result is processed to obtain the real-time data calculation result, and the specific steps include:

[0069] S41. Using the distributed engine, the real-time data association result is converted into a type according to a preset conversion rule to obtain a real-time data conversion result.

[0070] In step S41, the distributed engine is used to convert the original data extracted from the real-time data association result according to the preset conversion rules. The conversion rules are based on the needs of the insurance business and convert the data from one type to another type that conforms to the business logic, such as converting string data to integer or floating point type. Through this step, the system can ensure that the input data type conforms to the format required for subsequent calculations, and the result obtained is the real-time data conversion result.

[0071] S42: Filter the data that does not conform to the business logic in the real-time data conversion result based on a preset filtering rule to obtain a real-time data filtering result.

[0072] After the data type conversion is completed, the converted real-time data is filtered based on the preset filtering rules. The filtering rules are designed to eliminate data that does not conform to business logic or has abnormal values, such as negative premiums, repeated operation records, etc. The filtered data is the real-time data filtering result. This step ensures that the system only retains valid data that conforms to the insurance business logic when processing real-time data, reducing the impact of redundant data on system performance.

[0073] S43, a deduplication strategy based on a hash algorithm maps the real-time data filtering result to a unique identifier through a hash function and stores it in a distributed cache; when new data flows in, the unique identifier is quickly queried to determine whether the new data is a duplicate of the real-time data filtering result; if it is a duplicate, the new data is discarded; if it is not a duplicate, the new data is continued to be processed to obtain a real-time data deduplication result.

[0074] To further improve the accuracy of the data, a deduplication strategy based on a hash algorithm is used to deduplicate the filtered data. Specifically, a unique identifier is generated for each record in the real-time data filtering result through a hash function, and the unique identifier is stored in a distributed cache. When new data flows in, the system will quickly query the unique identifier to determine whether the new data already exists in the cache. If the new data is found to be duplicated with the data in the cache, the data is directly discarded; if the new data is not duplicated, it will continue to be processed to obtain the real-time data deduplication result.

[0075] S44, dynamically associate the real-time data deduplication result, and perform window aggregation calculation based on an aggregation mechanism to obtain the real-time data calculation result.

[0076] In step S44, the real-time data deduplication results are dynamically associated, and the real-time data is associated with other business data sources in real time. At the same time, window aggregation calculation is performed based on the aggregation mechanism, that is, all data within the specified time window is summarized and analyzed to obtain the final real-time data calculation result. Window aggregation can accumulate real-time data within a certain time interval, such as calculating the total premium or the number of orders within the time period. In this way, the system can provide dynamic analysis results of insurance business in real time to support business decision-making needs.

[0077] Through the above steps, the implementation method of the present invention effectively uses the distributed engine to process real-time data, ensuring the accuracy, consistency and timeliness of the data. The method can realize rapid processing and dynamic analysis of insurance data in high-concurrency business scenarios through type conversion, data filtering, deduplication and window aggregation, and is suitable for complex insurance data calculation scenarios.

[0078] Combination Figure 5 As shown, the real-time data calculation result is written into the real-time data table of the relational database to obtain the current period insurance accumulation data, and the specific steps include:

[0079] S45. Load the insurance coding rules from the standard insurance coding table and cache them into the memory.

[0080] In step S45, the system loads the insurance coding rules from the standard insurance coding table. The standard insurance coding table contains the coding specifications for various insurance businesses, such as product codes, policy codes, agency codes, life insurance codes, etc. These coding rules can ensure that different insurance products and business data have a unified coding standard in the system. After loading, the system caches these coding rules into memory to improve the efficiency of the subsequent matching process and avoid the need to reload each time an operation is performed.

[0081] S46. Match the real-time data calculation result with the insurance coding rule and replace it with the corresponding insurance coding value.

[0082] After the real-time data calculation results are generated, the system matches the business data in the real-time data calculation results with the insurance coding rules in the memory one by one. Each business data in the real-time data calculation results (such as insurance products, policies, etc.) will be replaced according to the insurance coding rules, replacing the business fields in the original data with standardized insurance coding values.

[0083] S47. Group the insurance code values ​​into batches and write them into the real-time data table of the relational database using a batch insert command.

[0084] In order to improve the efficiency of data writing, the system groups the replaced insurance code values ​​into batches according to preset rules. The grouped data will be processed in batches, and the system will write the data to the real-time data table of the relational database by executing batch insert commands (such as JDBC batch operations). Batch insert operations can reduce the number of database IO operations, improve write performance, and ensure data consistency, avoiding data conflicts or write failures caused by high concurrency.

[0085] In step S50, offline data of the previous cycle is obtained from the data operation layer in the data warehouse tool, and the offline data is calculated and processed using the data warehouse tool to obtain the calculation result of the offline data of the previous cycle.

[0086] Combination Figure 6 As shown, the offline data is obtained from the data operation layer in the data warehouse tool, and the specific steps include:

[0087] S51. Initialize the connection parameters of the data warehouse tool.

[0088] In step S51, the system needs to initialize the connection with the data warehouse tool. During the initialization process, parameters including the connection URL, authentication information, and timeout configuration of the data warehouse are set to ensure that the system can normally connect to the data operation layer of the data warehouse.

[0089] S52: Using the metadata management tool of the data warehouse tool to obtain the structural information of the data operation layer.

[0090] After successfully establishing the connection, the system uses the metadata management tool of the data warehouse tool to query and obtain the structural information of the data operation layer. The structural information includes the table structure, field information, data type and its corresponding index and relationship in the data operation layer.

[0091] S53: Start a query task through a computing cluster, extract the offline data in parallel, and cache the extracted offline data in a distributed file.

[0092] After obtaining the structural information of the data operation layer, the system starts the parallel query task through the computing cluster. Through the parallel computing capability of the computing cluster, the system can perform large-scale query operations in a short time and extract the required offline data. This parallel execution mechanism utilizes the distributed computing resources of the cluster, greatly improving the speed and efficiency of data extraction. The extracted offline data will be temporarily stored in distributed files.

[0093] In step S60, the offline data calculation results obtained by the above calculations are pushed to the offline data table in the relational database using a data transfer tool. The offline data table stores the accumulated data of the insurance business in the previous cycle, which is used for subsequent association operations with real-time data.

[0094] In step S70, the offline data table in the relational database is set as the main table, and a left association operation is performed on it and the real-time data table of the current cycle. Through this association operation, the real-time data of the current cycle and the offline data of the previous cycle are combined to generate complete insurance fusion data. The insurance fusion data not only contains the current real-time data, but also reflects the historical accumulated data, which is convenient for the business to conduct multi-dimensional comprehensive analysis.

[0095] Combination Figure 7 As shown, the offline data table is used as the main table and left-linked to the current period insurance accumulation data to obtain the insurance fusion data of the real-time data table and the offline data table. The specific steps include:

[0096] S71. Construct a left-association query statement based on the module association condition, and specify the main table as the offline data table and the secondary table as the current cycle insurance accumulated data.

[0097] In step S71, the system constructs a left-association query statement according to the preset module association condition. In the query statement, the offline data table is specified as the main table, and the current cycle insurance cumulative data is specified as the slave table. The module association condition is based on the logical rules of the insurance business, such as insurance product ID, policy ID, etc., and the association matching between the two tables can be achieved through these association fields. The left-association query method ensures that all data in the main table will be retained, and only the slave table data that meets the association condition will be matched to the main table.

[0098] S72. Use index technology to save all data in the main table, and match them with the data in the slave table that meets the module association conditions to obtain the insurance fusion data.

[0099] To improve the efficiency of data matching, the system will apply indexing technology in the master table. By indexing the data in the master table, the system can quickly locate records that meet the conditions, thereby accelerating the matching process with the data in the slave table. Indexing technology not only improves the efficiency of queries, but also reduces the overhead of full table scans, ensuring the performance of the system when processing large-scale data. In the process of executing left-association queries, the system uses indexing technology to match the data in the master table (offline data table) with the records in the slave table (current cycle insurance cumulative data) that meet the module association conditions. After a successful match, the system generates insurance fusion data, which contains complete information of offline data and real-time data.

[0100] Through the above steps, the system can effectively realize the fusion processing of real-time data tables and offline data tables, and generate fused data containing all insurance data. This process ensures the efficiency and completeness of data query through the combination of left-associative query statements and indexing technology, and is suitable for complex association calculation scenarios of insurance data. Through this implementation, the present invention uses indexing technology and left-associative query to optimize the efficiency of data processing in the process of data association and fusion, ensuring the performance and accuracy of the system when processing large-scale data, and is suitable for the data processing requirements of complex insurance business.

[0101] In step S80, a data view is created in the relational database based on the insurance fusion data generated above. The data view displays real-time data and offline data in a unified manner, and by binding with the preset API service, external applications can obtain the required insurance fusion data by calling the API. In this way, standardized data output can be achieved, and the external system can be guaranteed to obtain the latest insurance business data in real time, which enhances the flexibility and applicability of the system.

[0102] The present invention realizes the effective combination of real-time and offline calculation of insurance business data, and uses a distributed engine to ensure the efficiency and accuracy of data processing. This method not only improves the real-time and stability of the system, but also enhances the reusability and scalability of data, and can be widely used in data calculation scenarios of various insurance businesses.

[0103] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0104] In one embodiment, an insurance data calculation device based on a distributed engine is provided, and the insurance data calculation device based on a distributed engine corresponds one-to-one to the insurance data calculation method based on a distributed engine in the above embodiment. Figure 8 As shown, the insurance data computing device based on the distributed engine includes a log acquisition module 101, a log acquisition module 102, a log acquisition module 103, a real-time computing module 104, an offline computing module 105, a data migration module 106, a table association module 107 and a data output module 108. The functional modules are described in detail as follows:

[0105] The log acquisition module 101 is used to capture the insurance operation log of the relational database using structured data replication software, and write the insurance operation log into the topic of the distributed management platform;

[0106] The log acquisition module 102 is used to import the dimension logic table pre-calculated in the data warehouse tool into the relational database;

[0107] The log acquisition module 103 is used to acquire the insurance operation log from the subject of the distributed management platform by using a distributed engine, and associate it with the dimension logic table in real time to obtain a real-time data association result;

[0108] The real-time calculation module 104 is used to process the real-time data association result to obtain a real-time data calculation result, and write the real-time data calculation result into the real-time data table of the relational database to obtain the insurance accumulation data of the current period;

[0109] The offline calculation module 105 is used to obtain offline data from the data operation layer in the data warehouse tool, and use the data warehouse tool to calculate the insurance accumulation data of the previous period to obtain offline data calculation results;

[0110] A data migration module 106, used to push the offline data calculation results to the offline data table of the relational database through a data transfer tool;

[0111] Table association module 107, used to take the offline data table as the main table, and left-associate it to the current period insurance accumulation data, to obtain insurance fusion data of the real-time data table and the offline data table;

[0112] The data output module 108 is used to create a data view based on the relational database, and bind the data view to a preset API service to provide the insurance fusion data to external applications.

[0113] In one embodiment, the log acquisition module 101 is specifically used to:

[0114] identifying data operations in the relational database using the structured data replication software;

[0115] Extracting insert operations, update operations, and delete operations from the data operations, and generating corresponding log entries according to the operation types;

[0116] The log entries are formatted and stored in a log storage file in a preset structured format to obtain the insurance operation log.

[0117] In one embodiment, the log acquisition module 102 is specifically used to:

[0118] Designing a hierarchical structure of the dimension logic table according to preset business requirements to obtain a fact table and a dimension table;

[0119] Using a multidimensional data model, the fact table is logically associated with the dimension table to obtain the dimension logic table;

[0120] Based on the parallel computing framework of the data warehouse tool, a directed acyclic graph scheduling mechanism is used to define the execution order and dependency relationship of each computing task in the dimension logic table;

[0121] After the dimension logic table is calculated, different versions of the dimension logic table are partitioned and stored in the relational database.

[0122] In one embodiment, the real-time calculation module 104 is specifically used for:

[0123] Using the distributed engine to perform type conversion on the real-time data association result according to a preset conversion rule to obtain a real-time data conversion result;

[0124] Filtering the data that does not conform to the business logic in the real-time data conversion result based on the preset filtering rules to obtain the real-time data filtering result;

[0125] The deduplication strategy based on the hash algorithm maps the real-time data filtering result to a unique identifier through a hash function and stores it in a distributed cache; when new data flows in, the unique identifier is quickly queried to determine whether the new data is repeated with the real-time data filtering result; if repeated, the new data is discarded; if not repeated, the new data is continuously processed to obtain the real-time data deduplication result;

[0126] A dynamic association operation is performed on the real-time data deduplication result, and a window aggregation calculation is performed based on an aggregation mechanism to obtain the real-time data calculation result.

[0127] In one embodiment, the real-time calculation module 104 is further specifically used for:

[0128] Load insurance coding rules from the standard insurance coding table and cache them into memory;

[0129] Matching the real-time data calculation result with the insurance coding rule and replacing it with the corresponding insurance coding value;

[0130] The insurance code values ​​are grouped into batches and written into the real-time data table of the relational database using a batch insert command.

[0131] In one embodiment, the offline computing module 105 is specifically used for:

[0132] Initializing connection parameters of the data warehouse tool;

[0133] Using the metadata management tool of the data warehouse tool to obtain the structural information of the data operation layer;

[0134] The query task is started through the computing cluster, the extraction of the offline data is executed in parallel, and the extracted offline data is cached in a distributed file.

[0135] In one embodiment, the table association module 107 is specifically used to:

[0136] Construct a left-association query statement based on the module association condition, and specify the main table as the offline data table and the secondary table as the current period insurance accumulated data;

[0137] All data in the master table are saved by using index technology, and matched with data in the slave table that meets the module association conditions to obtain the insurance fusion data.

[0138] For the specific limitations of the insurance data calculation device based on the distributed engine, please refer to the limitations of the insurance data calculation method based on the distributed engine above, which will not be repeated here. Each module in the above-mentioned insurance data calculation device based on the distributed engine can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0139] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the service side of a distributed engine-based insurance data calculation method.

[0140] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a distributed engine-based insurance data calculation method.

[0141] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0142] Capturing the insurance operation log of the relational database using structured data replication software, and writing the insurance operation log into the topic of the distributed management platform;

[0143] Importing the dimension logic table calculated in advance in the data warehouse tool into the relational database;

[0144] Using a distributed engine to obtain the insurance operation log from the subject of the distributed management platform, and perform real-time association with the dimension logic table to obtain a real-time data association result;

[0145] Processing the real-time data association result to obtain a real-time data calculation result, and writing the real-time data calculation result into a real-time data table of the relational database to obtain the current period insurance accumulation data;

[0146] Obtain offline data from the data operation layer in the data warehouse tool, and use the data warehouse tool to calculate the insurance accumulation data of the previous period to obtain offline data calculation results;

[0147] Pushing the offline data calculation result to the offline data table of the relational database through a data transfer tool;

[0148] The offline data table is used as the main table, and is left-linked to the current period insurance accumulation data to obtain insurance fusion data of the real-time data table and the offline data table;

[0149] A data view is created based on the relational database, and the data view is bound to a preset API service to provide the insurance fusion data to external applications.

[0150] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0151] Capturing the insurance operation log of the relational database using structured data replication software, and writing the insurance operation log into the topic of the distributed management platform;

[0152] Importing the dimension logic table calculated in advance in the data warehouse tool into the relational database;

[0153] Using a distributed engine to obtain the insurance operation log from the subject of the distributed management platform, and perform real-time association with the dimension logic table to obtain a real-time data association result;

[0154] Processing the real-time data association result to obtain a real-time data calculation result, and writing the real-time data calculation result into a real-time data table of the relational database to obtain the current period insurance accumulation data;

[0155] Obtain offline data from the data operation layer in the data warehouse tool, and use the data warehouse tool to calculate the insurance accumulation data of the previous period to obtain offline data calculation results;

[0156] Pushing the offline data calculation result to the offline data table of the relational database through a data transfer tool;

[0157] The offline data table is used as the main table, and is left-linked to the current period insurance accumulation data to obtain insurance fusion data of the real-time data table and the offline data table;

[0158] A data view is created based on the relational database, and the data view is bound to a preset API service to provide the insurance fusion data to external applications.

[0159] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0161] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0162] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for calculating insurance data based on a distributed engine, characterized in that: include: Capturing the insurance operation log of the relational database using structured data replication software, and writing the insurance operation log into the topic of the distributed management platform; Importing the dimension logic table calculated in advance in the data warehouse tool into the relational database; Using a distributed engine to obtain the insurance operation log from the subject of the distributed management platform, and perform real-time association with the dimension logic table to obtain a real-time data association result; Processing the real-time data association result to obtain a real-time data calculation result, and writing the real-time data calculation result into a real-time data table of the relational database to obtain the current period insurance accumulation data; Obtain offline data from the data operation layer in the data warehouse tool, and use the data warehouse tool to calculate the insurance accumulation data of the previous period to obtain offline data calculation results; Pushing the offline data calculation result to the offline data table of the relational database through a data transfer tool; The offline data table is used as the main table, and is left-linked to the current period insurance accumulation data to obtain insurance fusion data of the real-time data table and the offline data table; A data view is created based on the relational database, and the data view is bound to a preset API service to provide the insurance fusion data to external applications.

2. The insurance data calculation method based on a distributed engine according to claim 1, characterized in that: The method of capturing the insurance operation log of the relational database using structured data replication software includes: identifying data operations in the relational database using the structured data replication software; Extracting insert operations, update operations, and delete operations from the data operations, and generating corresponding log entries according to the operation types; The log entries are formatted and stored in a log storage file in a preset structured format to obtain the insurance operation log.

3. The insurance data calculation method based on a distributed engine according to claim 1, characterized in that: The step of importing the dimension logic table pre-calculated in the data warehouse tool into the relational database includes: Designing a hierarchical structure of the dimension logic table according to preset business requirements to obtain a fact table and a dimension table; Using a multidimensional data model, the fact table is logically associated with the dimension table to obtain the dimension logic table; Based on the parallel computing framework of the data warehouse tool, a directed acyclic graph scheduling mechanism is used to define the execution order and dependency relationship of each computing task in the dimension logic table; After the dimension logic table is calculated, different versions of the dimension logic table are partitioned and stored in the relational database.

4. The insurance data calculation method based on a distributed engine according to claim 1, characterized in that: The performing data processing on the real-time data association result to obtain the real-time data calculation result includes: Using the distributed engine to perform type conversion on the real-time data association result according to a preset conversion rule to obtain a real-time data conversion result; Filtering the data that does not conform to the business logic in the real-time data conversion result based on the preset filtering rules to obtain the real-time data filtering result; The deduplication strategy based on the hash algorithm maps the real-time data filtering result to a unique identifier through a hash function and stores it in a distributed cache; when new data flows in, the unique identifier is quickly queried to determine whether the new data is repeated with the real-time data filtering result; if repeated, the new data is discarded; if not repeated, the new data is continuously processed to obtain the real-time data deduplication result; A dynamic association operation is performed on the real-time data deduplication result, and a window aggregation calculation is performed based on an aggregation mechanism to obtain the real-time data calculation result.

5. The insurance data calculation method based on a distributed engine according to claim 1, characterized in that: The step of writing the real-time data calculation result into the real-time data table of the relational database to obtain the current period insurance cumulative data includes: Load insurance coding rules from the standard insurance coding table and cache them into memory; Matching the real-time data calculation result with the insurance coding rule and replacing it with the corresponding insurance coding value; The insurance code values ​​are grouped into batches and written into the real-time data table of the relational database using a batch insert command.

6. The insurance data calculation method based on a distributed engine according to claim 1, characterized in that: The acquiring offline data from the data operation layer in the data warehouse tool includes: Initializing connection parameters of the data warehouse tool; Using the metadata management tool of the data warehouse tool to obtain the structural information of the data operation layer; The query task is started through the computing cluster, the extraction of the offline data is executed in parallel, and the extracted offline data is cached in a distributed file.

7. The insurance data calculation method based on a distributed engine according to claim 1, characterized in that: The offline data table is used as the main table and left-linked to the current period insurance accumulation data to obtain insurance fusion data of the real-time data table and the offline data table, including: Construct a left-association query statement based on the module association condition, and specify the main table as the offline data table and the secondary table as the current period insurance accumulated data; All data in the master table are saved by using index technology, and matched with data in the slave table that meets the module association conditions to obtain the insurance fusion data.

8. An insurance data computing device based on a distributed engine, characterized in that: include: A log acquisition module, used to capture the insurance operation log of the relational database using structured data replication software, and write the insurance operation log into the topic of the distributed management platform; A log acquisition module, used for importing the dimension logic table pre-calculated in the data warehouse tool into the relational database; A log acquisition module, used to acquire the insurance operation log from the subject of the distributed management platform by using a distributed engine, and associate it with the dimension logic table in real time to obtain a real-time data association result; A real-time calculation module, used for performing data processing on the real-time data association result to obtain a real-time data calculation result, and writing the real-time data calculation result into the real-time data table of the relational database to obtain the current period insurance accumulation data; An offline calculation module, used to obtain offline data from the data operation layer in the data warehouse tool, and use the data warehouse tool to calculate the insurance accumulation data of the previous period to obtain offline data calculation results; A data migration module, used to push the offline data calculation results to the offline data table of the relational database through a data transfer tool; A table association module is used to take the offline data table as the main table and left-associate it to the current period insurance accumulation data to obtain insurance fusion data of the real-time data table and the offline data table; A data output module is used to create a data view based on the relational database, and bind the data view to a preset API service to provide the insurance fusion data to an external application.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the insurance data calculation method based on the distributed engine as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the insurance data calculation method based on a distributed engine as described in any one of claims 1 to 7 are implemented.

Images