Data processing system based on big data

Abstract

The invention provides a data processing system based on big data, the system comprises a business end and a processing end, the processing end comprises an acquisition module, a storage module, a quality inspection module, a processing module, a scheduling module and a visualization module; the storage module is used for storing the structured data and the unstructured data in a preset database; the quality inspection module is used for acquiring the stored data in the preset database to perform quality inspection to determine an inspection condition; the processing module is used for determining whether data conversion processing needs to be carried out on the storage data or not according to the checking condition; the scheduling module is used for scheduling the storage data or the target data according to preset time to generate batch tasks; and the visualization module is used for performing visualization processing on the completed target batch running task according to the batch running condition to generate a target data table. The cost is reduced, the process is simplified, and the data volume processing effect of different requirements is met.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing system based on big data. Background Art

[0002] Currently, the databases commonly used by large companies in the insurance industry include Oracle Database, IBM Db, Teradata and SAP HANA. These database products have their own unique advantages in performance, scalability, security and enterprise support. Such high-end databases usually require powerful hardware support, such as high-performance servers, storage systems and network equipment. These hardware costs will also have a significant impact on the overall deployment cost. At the same time, database maintenance and support services are another important cost, especially when large-scale deployments are required or additional functions and support are required.

[0003] Databases commonly used by small and medium-sized insurance companies include open source databases such as MySQL and PostgreSQL. Their advantages are low cost, broad community support, and suitable for handling small and medium-sized data volumes and business needs. However, these databases may perform slightly worse when handling large-scale concurrency and complex queries, requiring more management and optimization work. As an open source database, MySQL has the advantages of low cost, easy deployment and maintenance, but its performance may be limited when handling large-scale concurrency and complex queries. SQLite is suitable for lightweight applications, but not suitable for high-concurrency writing and large-scale data processing. Microsoft SQL Server Express is free and easy to use, but it is limited by database size and hardware resources and is not suitable for handling large-scale data and complex business needs. In contrast, enterprise-level databases such as Oracle and IBM Db provide more powerful performance and support, but are also accompanied by high licensing fees and operation and maintenance costs, which may not be practical for small and medium-sized companies. Summary of the invention

[0004] In view of the above problems, the present application is proposed to provide a data processing system based on big data that overcomes the above problems or at least partially solves the above problems, including:

[0005] A data processing system based on big data, the system comprising a business end and a processing end, the processing end comprising a collection module, a storage module, a quality inspection module, a processing module, a scheduling module and a visualization module;

[0006] The acquisition module is used to collect data in the service end, wherein the data includes structured data and unstructured data;

[0007] The storage module is used to store both the structured data and the unstructured data in a preset database;

[0008] The quality inspection module is used to obtain the stored data in the preset database to perform quality inspection to determine the inspection status, wherein the inspection status includes pass and fail;

[0009] The processing module is used to determine whether the stored data needs to be converted according to the inspection situation, wherein when the inspection situation is passed, the stored data is converted to generate target data;

[0010] The scheduling module is used to schedule the stored data or the target data to generate a batch task according to a preset time, and monitor the batch task to determine the batch status, wherein the batch status includes completed and uncompleted; when the batch status is uncompleted, an early warning message is generated according to the current batch status;

[0011] The visualization module is used to generate a target data table for the completed target batch running task by performing visualization processing according to the batch running situation.

[0012] Furthermore, the scheduling module also includes a data platform monitoring submodule for monitoring and warning the data platform, wherein the data platform monitoring submodule includes a stream processing framework monitoring unit and a task scheduling system status monitoring unit;

[0013] The stream processing framework monitoring unit is used to determine the state information of the data stream through a preset program, and determine whether to generate the first alarm information through the state information and a preset alarm threshold, wherein the state information includes congestion, delay or loss;

[0014] The task scheduling system status monitoring unit is used to diagnose abnormal conditions on the data platform according to a set period through the preset program, and determine whether to generate a second alarm message based on the abnormal conditions, wherein the abnormal conditions include abnormal operation of the data flow and failure to complete the task on time.

[0015] Furthermore, the data platform monitoring submodule also includes a multi-channel notification unit, which is used to send the first alarm information or the second alarm information to the corresponding target user through a preset notification type after generating the first alarm information or the second alarm information, wherein the preset notification type includes telephone, communication software or email.

[0016] Furthermore, the multi-channel notification unit also includes an automatic reminder subunit, and the automatic reminder subunit sends the abnormality report corresponding to the first alarm information or the second alarm information to the target person through communication software.

[0017] Further, the acquisition module includes a query submodule, a performance monitoring submodule and a collection submodule;

[0018] The query submodule is used to query and analyze the execution plan corresponding to the collected data during the process of collecting data in the business terminal to determine the validity of the execution plan;

[0019] The performance monitoring submodule is used to perform real-time monitoring during the execution of the execution plan to ensure that the execution plan is in the optimal execution path;

[0020] The collection submodule is used to collect data in the business end when it is determined that the execution plan is valid and is in the optimal execution path.

[0021] Furthermore, the quality inspection module includes a standardization processing submodule, a missing value processing submodule, a duplicate data processing submodule, a consistency processing submodule, a cross-table verification submodule, a business rule verification submodule and a historical data comparison submodule;

[0022] The standardization processing submodule is used to perform standardization processing on the stored data with inconsistent formats;

[0023] The missing value processing submodule is used to select the corresponding missing value processing strategy for the stored data with missing values ​​according to the business logic;

[0024] The duplicate data processing submodule is used to identify and determine duplicate data in the stored data, and to clean up the duplicate data after identifying the duplicate data;

[0025] The consistency processing submodule is used to ensure the consistency of the policy data corresponding to the stored data between different tables or systems;

[0026] The cross-table verification submodule is used to determine the logical consistency of the stored data by presetting cross-table verification rules;

[0027] The business rule verification submodule is used to determine that the stored data complies with the business logic through preset business rules;

[0028] The historical data comparison submodule is used to obtain historical data, and compare the historical data with the stored data to determine whether there is an abnormal change in the stored data.

[0029] Furthermore, the visualization module includes a display template submodule, a display report submodule and a display chart submodule;

[0030] The display template submodule is used to generate a display template according to a preset blank template and the target batch task;

[0031] The display report submodule is used to generate a display report according to the target batch task through a report set by the user;

[0032] The display chart submodule is used to generate dynamic charts of different chart types according to the target batch running task, wherein the chart types include bar charts, line charts or pie charts.

[0033] Furthermore, the visualization module also includes a shielding submodule;

[0034] The shielding submodule is used to shield the privacy information corresponding to the display template, the display report and the dynamic chart respectively.

[0035] An embodiment of the present application further discloses a data processing system based on big data, the system comprising a business end and a processing end, the business end comprising an execution module and a data integration module;

[0036] The execution module is used to execute business processing according to business information to generate production data;

[0037] The data integration module is used to integrate the production data to generate business data, and the processing end collects data within the business data.

[0038] Further, the data integration module includes a data verification submodule and a missing check submodule;

[0039] The data verification submodule is used to perform field format verification or field range verification on production data according to preset custom business rules, wherein the preset custom business rules include regular expressions or conditional statements;

[0040] The missing check submodule is used to perform missing check on the required fields of the production data, mark the production data of the missing fields, and generate missing values ​​accordingly.

[0041] This application has the following advantages:

[0042] In the embodiments of the present application, the commonly used databases in the prior art may have slightly inferior performance when processing large-scale concurrency and complex queries, and require more management and optimization work. MySQL, as an open source database, has the advantages of low cost, easy deployment and maintenance, but its performance may be limited when processing large-scale concurrency and complex queries. SQLite is suitable for lightweight applications, but not suitable for high-concurrency writing and large-scale data processing. Microsoft SQL Server Express is free and easy to use, but is limited by database size and hardware resources and is not suitable for processing large-scale data and complex business needs. In contrast, enterprise-level databases such as Oracle and IBM Db provides more powerful performance and support, but it is also accompanied by high licensing fees and operation and maintenance costs, which may not be practical for small and medium-sized companies. The present application provides a solution that can save costs and simplify the query process, which is suitable for large-scale data reading and writing processing, specifically: a data processing system based on big data, the system includes a business end and a processing end, the processing end includes a collection module, a storage module, a quality inspection module, a processing module, a scheduling module and a visualization module; the collection module is used to collect data in the business end, wherein the data includes structured data and unstructured data; the storage module is used to store both the structured data and the unstructured data in a preset database; the quality inspection module is used to obtain the stored data in the preset database The quality inspection is performed according to the inspection status, wherein the inspection status includes pass and fail; the processing module is used to determine whether the stored data needs to be converted according to the inspection status, wherein when the inspection status is the pass, the stored data is converted to generate target data; the scheduling module is used to schedule the stored data or the target data according to the preset time to generate a batch task, and monitor the batch task to determine the batch status, wherein the batch status includes completed and uncompleted; when the batch status is the uncompleted, an early warning message is generated according to the current batch status; the visualization module is used to visualize the completed target batch task according to the batch status to generate a target data table. The processing end includes an acquisition module, a storage module, a quality inspection module, a processing module, a scheduling module and a visualization module to solve the problems of the existing database being expensive, complex query process, high data requirements and limited data volume, and achieve the effect of reducing costs, simplifying processes and meeting different needs of data volume processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 It is a structural block diagram of a data processing system based on big data provided by an embodiment of the present application;

[0045] Figure 2 It is a structural block diagram of a data processing system based on big data provided by an embodiment of the present application;

[0046] Figure 3 It is a data processing deployment diagram of a data processing system based on big data provided by an embodiment of the present application;

[0047] Figure 4 This is an overall architecture diagram of a data processing system based on big data provided by one embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objects, features and advantages of the present application more obvious and understandable, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0049] The inventors found through analysis of the prior art that:

[0050] The following solutions exist in the existing technical solutions: 1. Amazon Redshift + AWS Glue: However, the cost is high, especially in terms of data storage and query, which may lead to high operating costs. 2. Google BigQuery + Cloud Composer: However, it depends on the Google Cloud platform, and there may be data security and geographical restrictions, and the management complexity is relatively high.

[0051] 3. Microsoft Azure Synapse Analytics + Azure Data Factory: But it may be too complex and expensive for small companies and requires professional skills to configure and manage.

[0052] 4. Snowflake + Apache Airflow: However, it has high costs, especially in data storage and query, which may exceed the budget of small and medium-sized companies.

[0053] The present invention aims to solve the problems of real-time, accuracy and efficiency in the processing and analysis of insurance business data, and to improve the operating efficiency and data analysis capabilities of the data platform.

[0054] Reference Figure 1 , showing a structural block diagram of a data processing system based on big data provided by an embodiment of the present application;

[0055] A data processing system based on big data, the system comprising a business end and a processing end, the processing end comprising a collection module, a storage module, a quality inspection module, a processing module, a scheduling module and a visualization module;

[0056] The acquisition module is used to collect data in the service end, wherein the data includes structured data and unstructured data;

[0057] The storage module is used to store both the structured data and the unstructured data in a preset database;

[0058] The quality inspection module is used to obtain the stored data in the preset database to perform quality inspection to determine the inspection status, wherein the inspection status includes pass and fail;

[0059] The processing module is used to determine whether the stored data needs to be converted according to the inspection situation, wherein when the inspection situation is passed, the stored data is converted to generate target data;

[0060] The scheduling module is used to schedule the stored data or the target data to generate a batch task according to a preset time, and monitor the batch task to determine the batch status, wherein the batch status includes completed and uncompleted; when the batch status is uncompleted, an early warning message is generated according to the current batch status;

[0061] The visualization module is used to generate a target data table for the completed target batch running task by performing visualization processing according to the batch running situation.

[0062] In the embodiments of the present application, the commonly used databases in the prior art may have slightly inferior performance when processing large-scale concurrency and complex queries, and require more management and optimization work. MySQL, as an open source database, has the advantages of low cost, easy deployment and maintenance, but its performance may be limited when processing large-scale concurrency and complex queries. SQLite is suitable for lightweight applications, but not suitable for high-concurrency writing and large-scale data processing. Microsoft SQL Server Express is free and easy to use, but is limited by database size and hardware resources and is not suitable for processing large-scale data and complex business needs. In contrast, enterprise-level databases such as Oracle and IBM Db provides more powerful performance and support, but it is also accompanied by high licensing fees and operation and maintenance costs, which may not be practical for small and medium-sized companies. The present application provides a solution that can save costs and simplify the query process, which is suitable for large-scale data reading and writing processing, specifically: a data processing system based on big data, the system includes a business end and a processing end, the processing end includes a collection module, a storage module, a quality inspection module, a processing module, a scheduling module and a visualization module; the collection module is used to collect data in the business end, wherein the data includes structured data and unstructured data; the storage module is used to store both the structured data and the unstructured data in a preset database; the quality inspection module is used to obtain the stored data in the preset database The quality inspection is performed according to the inspection status, wherein the inspection status includes pass and fail; the processing module is used to determine whether the stored data needs to be converted according to the inspection status, wherein when the inspection status is the pass, the stored data is converted to generate target data; the scheduling module is used to schedule the stored data or the target data according to the preset time to generate a batch task, and monitor the batch task to determine the batch status, wherein the batch status includes completed and uncompleted; when the batch status is the uncompleted, an early warning message is generated according to the current batch status; the visualization module is used to visualize the completed target batch task according to the batch status to generate a target data table. The processing end includes an acquisition module, a storage module, a quality inspection module, a processing module, a scheduling module and a visualization module to solve the problems of the existing database being expensive, complex query process, high data requirements and limited data volume, and achieve the effect of reducing costs, simplifying processes and meeting different needs of data volume processing.

[0063] Next, a data processing system based on big data in this exemplary embodiment will be further described.

[0064] It should be noted that Greenplum is a type of distributed database. Its architecture adopts MPP (massive parallel processing). It uses powerful parallel processing capabilities and resource queue management to implement resource allocation by user groups. By building a cluster, it can group, store and analyze data ranging from TB to PB.

[0065] DolphinScheduler is a distributed big data workflow scheduling system developed in Java. It supports the definition of complex workflows through a graphical interface or JSON / YAML files, and can manage various types of tasks such as Shell, Python, Spark, Hive SQL, etc. The platform provides a visual interface to monitor the task status, and supports task dependency management, alarm notifications, role-based permission control, cluster management, multi-tenant support and other functions. It is suitable for scenarios such as data pipeline management and batch processing tasks.

[0066] Oracle GoldenGate (OGG) is a high-performance data replication and integration software that can capture database changes in real time and transmit these changes to other databases or data warehouses, supporting data migration and synchronization in heterogeneous environments.

[0067] Alibaba Canal is a lightweight data synchronization tool that is mainly used to capture binlog logs from MySQL databases and publish these changed data in the form of messages. It supports real-time data synchronization to other systems or data warehouses.

[0068] Apache Kafka is a distributed stream processing platform that can efficiently process and transmit large amounts of real-time data streams, supports high-throughput message publishing and subscription modes, and is often used to build real-time data pipelines and microservice architectures.

[0069] Apache Flink is an open source stream processing framework for processing unbounded and bounded data streams. It supports low-latency real-time data processing and event time processing, and is suitable for building high-performance data stream processing and analysis applications.

[0070] Python is a high-level programming language known for its concise syntax and powerful functions. It is widely used in web development, data science, artificial intelligence, automated scripts and other fields. It supports multiple programming paradigms such as object-oriented, imperative and functional programming.

[0071] In one embodiment of the present invention, the scheduling module also includes a data platform monitoring submodule for monitoring and warning the data platform, wherein the data platform monitoring submodule includes a stream processing framework monitoring unit and a task scheduling system status monitoring unit.

[0072] It should be noted that for credit guarantee insurance business, the timeliness of data flow processing and the on-time completion of tasks are crucial, especially when processing key data such as policy information, claims recovery, etc., it is necessary to ensure the stability and reliability of the system.

[0073] In one embodiment of the present invention, the stream processing framework monitoring unit is used to determine the status information of the data flow through a preset program, and determine whether to generate a first alarm information through the status information and a preset alarm threshold, wherein the status information includes congestion, delay or loss; in a specific embodiment, the Python program, i.e., the preset program, can monitor the status of the data flow by integrating Kafka's API, and regularly check whether the data flow is congested, delayed or lost; for example, Python calls Kafka's kafka-python library, monitors the data reception through the Kafka object, and sets a preset alarm threshold. If it is greater than the preset alarm threshold, a first alarm information is generated.

[0074] In one embodiment of the present invention, the task scheduling system status monitoring unit is used to diagnose abnormal conditions of the data platform according to a set period through the preset program, and determine whether to generate a second alarm information based on the abnormal conditions, wherein the abnormal conditions include abnormal operation of the data flow and failure to complete the task on time; in a specific embodiment, through the DolphinScheduler scheduling platform, a monitoring program written in Python, i.e., a preset program, is used to periodically scan the operating status and abnormal conditions of the data platform, including whether the data flow is normal, whether the task is completed on time, etc., to monitor the time and status of task execution. If the task is not completed within the predetermined time window, an alarm is triggered to generate a second alarm information.

[0075] In one embodiment of the present invention, the data platform monitoring submodule also includes a multi-channel notification unit, which is used to send the first alarm information or the second alarm information to the corresponding target user through a preset notification type after generating the first alarm information or the second alarm information, wherein the preset notification type includes telephone, communication software or email; in a specific embodiment, the Python program supports multiple warning methods such as telephone, enterprise WeChat, email, etc. by integrating different communication APIs; when the monitoring program detects an abnormal situation, the system will automatically trigger a pre-set warning notification and notify the operation and maintenance team in time so that they can respond and handle problems quickly to ensure the normal operation of the data platform.

[0076] It should be noted that credit guarantee insurance business involves high-risk data processing. Once a problem occurs in the system, relevant personnel need to be notified quickly to respond.

[0077] In one embodiment of the present invention, the multi-channel notification unit further includes an automatic reminder subunit, which sends the abnormal report corresponding to the first alarm information or the second alarm information to the target personnel through the communication software; in a specific embodiment, the Python pandas library is used to generate data statistics, and the automatic push is realized in combination with the enterprise WeChat API; automatic report push and operation and maintenance status feedback are realized. In another embodiment, the monitoring program can also automatically send key operating indicators and abnormal reports to the target personnel through digital employee integration to improve operation and maintenance efficiency and response speed; it can improve the operation and maintenance efficiency of the company's credit insurance business, automatically generate and push reports, and reduce manual intervention.

[0078] In one embodiment of the present invention, the acquisition module includes a query submodule, a performance monitoring submodule and a acquisition submodule.

[0079] It should be noted that SQL query plans should be reviewed regularly to ensure that the query execution path is optimal and to improve resource utilization efficiency.

[0080] In one embodiment of the present invention, the query submodule is used to perform query analysis on the execution plan corresponding to the collected data during the process of collecting data in the business terminal to determine the validity of the execution plan; in a specific embodiment, the EXPLAIN command is used to view the query execution plan, analyze each step of the query, and whether the index is effectively utilized.

[0081] In one embodiment of the present invention, the performance monitoring submodule is used to perform real-time monitoring during the execution of the execution plan to ensure that the execution plan is in the optimal execution path; in a specific embodiment, a Greenplum monitoring management platform such as Pivotal Greenplum Command Center (GPCC) and other tools are used to monitor CPU, memory, IO, network and other information in real time to identify performance bottlenecks.

[0082] In one embodiment of the present invention, the collection submodule is used to collect data within the business end when it is determined that the execution plan is valid and is on the optimal execution path; in a specific embodiment, data is collected from the business end when the execution plan is valid and on the optimal execution path.

[0083] In one embodiment of the present invention, the quality inspection module includes a standardization processing submodule, a missing value processing submodule, a duplicate data processing submodule, a consistency processing submodule, a cross-table verification submodule, a business rule verification submodule and a historical data comparison submodule.

[0084] It should be noted that data quality issues should be automatically detected and fixed, and data validation logic should be designed to ensure data consistency and accuracy.

[0085] In one embodiment of the present invention, the standardization processing submodule is used to perform standardization processing on the stored data with inconsistent formats; in a specific embodiment, standardization processing is performed on data with inconsistent formats.

[0086] In one embodiment of the present invention, the missing value processing submodule is used to filter the corresponding missing value processing strategy for the stored data with missing values ​​according to the business logic; in a specific embodiment, a suitable missing value processing strategy is selected according to the business logic. For example, if the loan number is missing, first determine whether the policy number is generated in the registration information table. If not, delete the data; if it has been generated, retain the data and wait for the number to be supplemented.

[0087] In one embodiment of the present invention, the duplicate data processing submodule is used to identify and determine duplicate data in the stored data, and clean up the duplicate data after identifying the duplicate data; in a specific embodiment, duplicate data is automatically identified and cleaned up. For duplicate business number records, duplicate items can be merged or deleted according to business rules (such as the most recently updated record).

[0088] In one embodiment of the present invention, the consistency processing submodule is used to ensure the consistency of the policy data corresponding to the stored data between different tables or systems; in a specific embodiment, the consistency of the same policy data between different tables or systems is ensured; for example, the consistency of the customer's premium amount in the business system and the payment system is verified.

[0089] In one embodiment of the present invention, the cross-table verification submodule is used to determine the logical consistency of the stored data by preset cross-table verification rules; in a specific embodiment, cross-table verification rules are set to ensure the logical consistency of the data; for example, verify that the policy number in the recovery information table does exist in the claim form, policy information table and registration information table to ensure the integrity of the associated data.

[0090] In one embodiment of the present invention, the business rule verification submodule is used to determine whether the stored data complies with the business logic through preset business rules; in a specific embodiment, the data is ensured to comply with the business logic through customized business rules; for example, verifying whether the premium calculation in the credit guarantee insurance is correct and ensuring that the ratio between the loan amount and the insurance amount complies with business regulations.

[0091] In one embodiment of the present invention, the historical data comparison submodule is used to obtain historical data, and compare the historical data with the stored data to determine whether there are abnormal changes in the stored data; in a specific embodiment, abnormal changes are detected by comparing historical data and current data; for example, if the underwriting balance of a channel business fluctuates abnormally in a short period of time, the system will automatically trigger an alarm and conduct an investigation.

[0092] In one embodiment of the present invention, the visualization module includes a display template submodule, a display report submodule and a display chart submodule;

[0093] In one embodiment of the present invention, the display template submodule is used to generate a display template based on a preset blank template and the target batch task; in a specific embodiment, a predefined report template is provided, and controls such as drop-down menus and sliders are provided to enhance the interactivity of the report and facilitate users to quickly obtain required information.

[0094] In one embodiment of the present invention, the display report submodule is used to generate a display report based on the target batch task through a report set by the user; in a specific embodiment, it supports users to define the layout and content of the report to meet personalized needs.

[0095] In one embodiment of the present invention, the display chart submodule is used to generate dynamic charts of different chart types based on the target batch running task, wherein the chart types include bar charts, line charts or pie charts; in a specific embodiment, multiple chart types are supported, such as bar charts, line charts, pie charts, etc., and the view can be customized through functions such as dragging and dropping, zooming and filtering, and dynamic updating of charts is supported.

[0096] In one embodiment of the present invention, the visualization module further includes a shielding submodule;

[0097] The shielding submodule is used to shield the privacy information corresponding to the display template, the display report and the dynamic chart respectively; in a specific embodiment, sensitive information such as personal identity information is automatically hidden during the data query process.

[0098] In one embodiment of the present invention, it also includes a data partitioning and index optimization module, which performs storage partitioning and index optimization according to the query frequency and query data scale of the policy information and overdue information. Among them, the overdue information table has a large amount of data and involves many fields, so multi-layer partitioning is used to improve data query efficiency; in a specific embodiment, for the policy information, a date-based partitioning strategy is mainly used. By using PARTITION BY and SUBPARTITION (product, channel number, date), data is effectively and evenly distributed to avoid performance bottlenecks during query; a multi-column composite index is created for the frequently queried fields of the overdue information table (such as overdue date, channel source, region, etc.).

[0099] In one embodiment of the present invention, it also includes an intelligent data compression module, which uses differentiated compression methods according to the characteristics of business data, and configures different window sizes and compression strategies; in a specific embodiment, the ZLIB compression library is used, and the bottom layer is the DEFLATE algorithm. ZLIB provides 10 compression levels (0-9) of compression control, from the fastest speed to the highest compression rate. By adjusting the compression level to control the compression rate and compression speed, the storage space requirement is reduced without sacrificing the reading performance, and the best effect is obtained. For example, the monthly aging report data has a low refresh frequency and a large storage space, and level 9 (the highest compression rate but the slowest speed) is used; the overdue information table data has a high refresh frequency and a moderate storage space, and level 5 is used to balance the processing speed and compression effect.

[0100] In one embodiment of the present invention, it also includes an intelligent batch processing and scheduled task module to manage the refresh frequency of the accumulated account age data and the task execution time; in a specific embodiment, the refresh frequency and task execution time are changed after the accumulated account age data reaches a certain level, and are managed through scheduled tasks and batch jobs, and data update tasks are executed regularly to avoid manual operations and the risk of data inconsistency.

[0101] In one embodiment of the present invention, it also includes an intelligent data loading and storage application module, which adjusts the storage level to improve performance according to the frequency of use of business data, the degree of resource loss, and importance; in a specific embodiment, a temporary table is used for intermediate result storage and data analysis, and the stored procedure encapsulates complex data processing logic to reduce network transmission and data processing time. At the same time, a "scheduled task" is used to run shell scripts regularly to achieve conditional deletion of temporary tables, which not only avoids excessive resource occupation by temporary tables, but also improves the maintainability and reusability of the code; efficient data loading tools and custom scripts are used to optimize the data loading and storage process, automatically adjust the data storage level according to the frequency and importance of data use, and adjust the allocation of computing resources to optimize performance.

[0102] In one embodiment of the present invention, it also includes an intelligent data lifecycle management module to manage the lifecycle of different data in the credit guarantee insurance business and reduce storage costs; for example, the overdue and recovered data in the later stage of the product is less important than that in the middle stage of the product, and the recovered data in the later stage of the product can be compressed and simplified and archived to reduce storage costs; in a specific embodiment, automatic archiving rules are set, and historical data that is not frequently accessed is regularly simplified or migrated to cold storage for archiving based on conditions such as the timestamp, access frequency, and data type of the data; the data lifecycle is set in combination with business needs and data compliance requirements, and the archiving operation is automatically triggered after the data reaches the retention period.

[0103] Reference Figure 2 , showing a structural block diagram of a data processing system based on big data provided by an embodiment of the present application;

[0104] A data processing system based on big data, the system comprising a business end and a processing end, the business end comprising an execution module and a data integration module;

[0105] The execution module is used to execute business processing according to business information to generate production data;

[0106] The data integration module is used to integrate the production data to generate business data, and the processing end collects data within the business data.

[0107] In one embodiment of the present invention, the data integration module includes a data verification submodule and a missing check submodule;

[0108] It should be noted that the data processing of credit guarantee insurance involves multiple complex channels, and the channel data access standards are different. When the data is transmitted from the credit insurance system to the middle platform, problems such as data inconsistency, missing, duplication or format errors will occur.

[0109] The data verification submodule is used to perform field format verification or field range verification on production data according to preset custom business rules, wherein the preset custom business rules include regular expressions or conditional statements; in a specific embodiment, regular expressions and custom business rules are used to verify the format and range of data fields; for example, regular expressions are used to check whether the recovery amount is less than the claim amount, and conditional statements are used to determine whether the loan amount in the loan information is within a reasonable range.

[0110] The missing check submodule is used to perform missing check on required fields of production data, mark the production data with missing fields, and generate missing values ​​accordingly; in a specific embodiment, for important required fields, such as policy number, promissory note number, etc., the data table is scanned regularly to identify and mark records with missing values.

[0111] Embodiment 1

[0112] like Figure 3 and Figure 4 As shown:

[0113] The data warehouse is built based on the distributed database ClickHouse, which supports large-scale data storage and high-concurrency queries.

[0114] Real-time data entry and data collection: Use Oracle GoldenGate (ogg) or Alibaba Canal to enter data into the lake in real time to ensure the timeliness and integrity of the data.

[0115] Message middleware: Apache Kafka is introduced as the message middleware to achieve reliable data transmission and decoupling, and support high-throughput data stream processing.

[0116] Streaming data processing: Combined with Apache Flink, it implements complex streaming data processing and real-time computing, supporting real-time business analysis and decision-making.

[0117] Intelligent scheduling task management: The batch task scheduling of the data platform is completed by DolphinScheduler, which realizes the automation and reliability of the data processing process through its flexible task scheduling and dependency management functions. Use scheduled tasks and triggers to arrange regular data processing work, and realize task scheduling based on conditions (such as starting processing only when specific data arrives).

[0118] Beneficial effects of this application:

[0119] Improve the real-time and accuracy of insurance business data processing and analysis, and achieve instant data processing and accurate analysis through scripting skills and optimization algorithms. Reduce the operation and maintenance costs of the data platform, simplify the complexity of the system, and use automated scripts and containerization technology to reduce manual intervention and management costs. Improve system response speed and data processing efficiency, optimize storage and query through scripts, and improve the efficiency of data access and analysis. Improve the scalability and adaptability of the data platform, use scripts to achieve rapid changes and expansion of the system, and support rapid response and expansion of future business needs.

[0120] By meeting the development needs of the credit insurance business, an efficient and scalable data processing and storage data warehouse has been built and implemented. This architecture can support big data management and efficient data processing, ensuring the rapid access and processing capabilities of credit insurance data. The data processing speed and system response time of the credit insurance business have been significantly improved, allowing the business team to obtain the required insights faster and support decision-making.

[0121] In view of the characteristics of credit insurance business, we have developed a comprehensive data processing and analysis model, including but not limited to data partitioning, index optimization, intelligent compression, batch management and other strategies. At the same time, we have realized the intelligent application of functions such as data lifecycle management, data cleaning and verification, which has significantly improved data processing efficiency and accuracy, reduced storage costs, and enabled insurance companies to more accurately assess risks and price products.

[0122] It also creates an interactive intelligent data warehouse that supports dynamic generation and customized views, greatly enhancing the flexibility and practicality of data visualization. It can quickly generate required reports and charts according to needs, greatly simplifying the data analysis process and improving the work efficiency and decision-making quality of business personnel.

[0123] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0124] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0125] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0126] The above is a detailed introduction to a data processing system based on big data provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing system based on big data, characterized in that: The system includes a business end and a processing end, wherein the processing end includes a collection module, a storage module, a quality inspection module, a processing module, a scheduling module and a visualization module; The acquisition module is used to collect data in the service end, wherein the data includes structured data and unstructured data; The storage module is used to store both the structured data and the unstructured data in a preset database; The quality inspection module is used to obtain the stored data in the preset database to perform quality inspection to determine the inspection status, wherein the inspection status includes pass and fail; The processing module is used to determine whether the stored data needs to be converted according to the inspection situation, wherein when the inspection situation is passed, the stored data is converted to generate target data; The scheduling module is used to schedule the stored data or the target data to generate a batch task according to a preset time, and monitor the batch task to determine the batch status, wherein the batch status includes completed and uncompleted; when the batch status is uncompleted, an early warning message is generated according to the current batch status; The visualization module is used to generate a target data table for the completed target batch running task by performing visualization processing according to the batch running situation.

2. The system according to claim 1, characterized in that The scheduling module also includes a data platform monitoring submodule for monitoring and warning the data platform, wherein the data platform monitoring submodule includes a stream processing framework monitoring unit and a task scheduling system status monitoring unit; The stream processing framework monitoring unit is used to determine the state information of the data stream through a preset program, and determine whether to generate the first alarm information through the state information and a preset alarm threshold, wherein the state information includes congestion, delay or loss; The task scheduling system status monitoring unit is used to diagnose abnormal conditions on the data platform according to a set period through the preset program, and determine whether to generate a second alarm message based on the abnormal conditions, wherein the abnormal conditions include abnormal operation of the data flow and failure to complete the task on time.

3. The system according to claim 2, characterized in that The data platform monitoring submodule also includes a multi-channel notification unit, which is used to send the first alarm information or the second alarm information to the corresponding target user through a preset notification type after generating the first alarm information or the second alarm information, wherein the preset notification type includes telephone, communication software or email.

4. The system according to claim 3, characterized in that The multi-channel notification unit further includes an automatic reminder subunit, which sends the abnormality report corresponding to the first alarm information or the second alarm information to the target person through communication software.

5. The system according to claim 1, characterized in that The acquisition module includes a query submodule, a performance monitoring submodule and a collection submodule; The query submodule is used to query and analyze the execution plan corresponding to the collected data during the process of collecting data in the business terminal to determine the validity of the execution plan; The performance monitoring submodule is used to perform real-time monitoring during the execution of the execution plan to ensure that the execution plan is in the optimal execution path; The collection submodule is used to collect data in the business end when it is determined that the execution plan is valid and is in the optimal execution path.

6. The system according to claim 1, characterized in that The quality inspection module includes a standardization processing submodule, a missing value processing submodule, a duplicate data processing submodule, a consistency processing submodule, a cross-table verification submodule, a business rule verification submodule and a historical data comparison submodule; The standardization processing submodule is used to perform standardization processing on the stored data with inconsistent formats; The missing value processing submodule is used to select the corresponding missing value processing strategy for the stored data with missing values ​​according to the business logic; The duplicate data processing submodule is used to identify and determine duplicate data in the stored data, and to clean up the duplicate data after identifying the duplicate data; The consistency processing submodule is used to ensure the consistency of the policy data corresponding to the stored data between different tables or systems; The cross-table verification submodule is used to determine the logical consistency of the stored data by presetting cross-table verification rules; The business rule verification submodule is used to determine that the stored data complies with the business logic through preset business rules; The historical data comparison submodule is used to obtain historical data, and compare the historical data with the stored data to determine whether there is an abnormal change in the stored data.

7. The system according to claim 1, characterized in that The visualization module includes a display template submodule, a display report submodule and a display chart submodule; The display template submodule is used to generate a display template according to a preset blank template and the target batch task; The display report submodule is used to generate a display report according to the target batch task through a report set by the user; The display chart submodule is used to generate dynamic charts of different chart types according to the target batch running task, wherein the chart types include bar charts, line charts or pie charts.

8. The system according to claim 7, characterized in that The visualization module also includes a shielding submodule; The shielding submodule is used to shield the privacy information corresponding to the display template, the display report and the dynamic chart respectively.

9. A data processing system based on big data, characterized in that: The system includes a business end and a processing end, and the business end includes an execution module and a data integration module; The execution module is used to execute business processing according to business information to generate production data; The data integration module is used to integrate the production data to generate business data, and the processing end collects data within the business data.

10. The system according to claim 9, characterized in that The data integration module includes a data verification submodule and a missing check submodule; The data verification submodule is used to perform field format verification or field range verification on production data according to preset custom business rules, wherein the preset custom business rules include regular expressions or conditional statements; The missing check submodule is used to perform missing check on the required fields of the production data, mark the production data of the missing fields, and generate missing values ​​accordingly.

Images