A multi-dimension data real-time statistical method and device based on multi-table change trajectories

Abstract

The application discloses a kind of multi-dimension data real-time statistical method and device based on multiple table change trajectory, the method includes: obtaining inventory data from relational database Informix, inventory data is batched by ETL, and the inventory data after processing is stored to Redis database;Incremental data is obtained from Kafka platform, and the incremental data is streamed by Flink, and the incremental data after processing is stored to Redis database;Update data in incremental data is obtained, and update data is batched, and the update data after processing is stored to Redis database;The inventory data after processing, incremental data and update data are obtained from Redis database, and are summarized according to different dimensions, and the summary statistical result is updated in terminal application in real time. The application solves the problem that query efficiency is slow and business data statistics is not real-time due to the growth of business year by year.

Description

Technical Field

[0001] This invention relates to the field of real-time data statistics, and in particular to a method and apparatus for real-time statistics of multi-dimensional data based on the change trajectory of multiple tables. Background Technology

[0002] When querying and statistically analyzing business data, existing solutions are generally based on the traditional business Informix database. Incremental data in the business database is written to a text file, processed by traditional ETL tools, and then merged with the existing data to finally achieve a summary for front-end report display.

[0003] As the business gradually developed, the following defects emerged:

[0004] 1. The amount of data in the traditional Informix database source data end is increasing year by year, the query efficiency is getting slower and slower, and the frequency and risk of data unloading errors are increasing;

[0005] 2. Frequent queries to the Informix source database increase the operational pressure on the source database.

[0006] 3. It cannot achieve true real-time data statistics and cannot meet actual business needs. Summary of the Invention

[0007] This invention provides a method and apparatus for real-time statistical analysis of multi-dimensional data based on the change trajectory of multiple tables, which solves the problems of slow query efficiency and low real-time performance of business data statistics caused by the year-on-year growth of business.

[0008] A real-time statistical method for multi-dimensional data based on multi-table change trajectories includes:

[0009] Retrieve existing data from the relational database Informix, perform batch processing on the existing data using ETL, and store the processed existing data in the Redis database;

[0010] Incremental data is obtained from the Kafka platform, stream-processed using Flink, and then stored in the Redis database.

[0011] Obtain the updated data from the incremental data, perform batch processing on the updated data, and store the processed updated data in the Redis database;

[0012] The system retrieves processed existing data, incremental data, and updated data from the Redis database, summarizes and statistically analyzes them according to different dimensions, and displays the summarized statistical results in the terminal application in real time.

[0013] In one embodiment of the present invention, the step of obtaining existing data from the relational database Informix and performing batch processing on the existing data via ETL specifically includes: obtaining payment and receipt reason data and data within a statistical time range from the Informix database, converting the data into a text file via ETL; storing the text file in an intermediate relational database Oracle; summarizing the text file stored in the Oracle database by associating it with table information of different dimensions; periodically updating the summarized data to the Oracle result table based on a preset frequency; and loading and updating the data in the Oracle result table to the Redis database via a Flink task workflow.

[0014] In one embodiment of the present invention, the incremental data stream processing via Flink specifically includes: consuming relevant Topic data from the Kafka platform, filtering out incremental data within a fixed payment / receipt reason and payment / receipt date range, and storing the filtered incremental data in a relational database PostgreSQL; using the incremental data as the main data stream, loading table information of different dimensions into dimension table data streams; sequentially associating the main data stream and the dimension table data streams; aggregating the associative result set according to different dimensions, merging the aggregation results belonging to the same dimension, and storing the merged dataset in a Redis database.

[0015] In one embodiment of the present invention, the step of obtaining updated data from the incremental data and batch processing the updated data specifically includes: obtaining updated data from the incremental data stored in PostgreSQL; determining historical newly added data corresponding to the updated data through primary key association; multiplying the amount in the historical newly added data by a preset value, and then merging and summarizing it with the updated data.

[0016] In one embodiment of the present invention, the step of converting the data into a text file through ETL processing specifically includes: cleaning the data obtained from the Informix database, including deleting invalid data, filling missing values, and removing duplicate records; and converting the format of the cleaned data to obtain a text file.

[0017] In one embodiment of the present invention, the method further includes: monitoring the logs of the stream processing and batch processing processes, and obtaining log data during the stream processing and batch processing processes; performing multidimensional analysis on the logs, and determining whether there are any anomalies based on the results of the multidimensional analysis; and automatically sending error information of the batch processing or stream processing to relevant personnel when an anomaly is detected in the logs.

[0018] In one embodiment of the present invention, the method further includes: enabling a checkpoint mechanism and configuring checkpoints in a Flink task, configuring a restart strategy when a task fails; capturing possible exceptions, triggering a Flink task failure when an exception is captured, and marking the failed Flink task; querying the most recent checkpoint of the marked Flink task, re-executing the failed task based on the data status of the most recent checkpoint, and resuming from the last saved checkpoint.

[0019] A real-time statistical device for multi-dimensional data based on multi-table change trajectories includes:

[0020] The existing data processing module is used to obtain existing data from the relational database Informix, perform batch processing on the existing data through ETL, and store the processed existing data in the Redis database;

[0021] The incremental data processing module is used to obtain incremental data from the Kafka platform, perform stream processing on the incremental data through Flink, and store the processed incremental data in the Redis database.

[0022] An update data processing module is used to obtain the updated data from the incremental data, perform batch processing on the updated data, and store the processed updated data in a Redis database.

[0023] The summary and statistics module is used to retrieve the processed existing data, incremental data, and updated data from the Redis database, and perform summary statistics according to different dimensions, and update and display the summary and statistics results in the terminal application in real time.

[0024] A real-time statistical device for multi-dimensional data based on multi-table change trajectories includes:

[0025] At least one processor; and,

[0026] The memory is communicatively connected to the at least one processor via a bus; wherein,

[0027] The memory stores instructions that can be executed by the at least one processor to implement the method as described in any of the above embodiments.

[0028] A non-volatile storage medium storing computer-executable instructions, which are executed by a processor to implement the method as described in any of the above embodiments.

[0029] This invention provides a method and apparatus for real-time statistical analysis of multi-dimensional data based on multi-table change trajectories, which has at least the following beneficial effects:

[0030] 1. Using the company's full data Kafka platform as the incremental data source reduces the database computation pressure of incremental data batch processing and reduces the risk of gradually increasing incremental data query efficiency;

[0031] 2. Reduced the pressure on the company's Informix database for real-time data queries;

[0032] 3. By using Flink streaming data processing technology, the frequency of real-time data processing has been greatly improved, meeting the actual needs of higher business frequencies. Attached Figure Description

[0033] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0034] Figure 1 This is a flowchart of real-time statistics of multi-table data based on Flink, provided in an embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram illustrating the steps of a real-time statistical method for multi-dimensional data based on multi-table change trajectories provided in an embodiment of the present invention.

[0036] Figure 3 A schematic diagram of a multi-dimensional data real-time statistical device based on multi-table change trajectories provided in an embodiment of the present invention;

[0037] Figure 4 This is a schematic diagram of a multi-dimensional data real-time statistical device based on the change trajectory of multiple tables, provided as an embodiment of the present invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be described clearly and completely below in conjunction with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0039] It should be noted that those skilled in the art will understand, explicitly and implicitly, that the embodiments described in this invention can be combined with other embodiments without conflict. Unless otherwise defined, the technical or scientific terms used in this invention should be understood in their ordinary sense by those skilled in the art. The terms "a," "an," "an," "the," etc., used in this invention do not indicate quantity limitation and can represent singular or plural. The terms "comprising," "including," "having," and any variations thereof used in this invention are intended to cover non-exclusive inclusion; the terms "first," "second," "third," etc., used in this invention are merely to distinguish similar objects and do not represent a specific ordering of objects.

[0040] The following explains some of the concepts involved in this application.

[0041] ETL (Extract-Transform-Load) is an abbreviation for Extract, Transform, and Load, which follows the process of loading data from a source system into a data warehouse. This process includes the following steps:

[0042] Extraction: The main goal of this stage is to aggregate multiple data sources in preparation for the next step of transformation.

[0043] Transformation: This is the core step of ETL. Its main goal is to clean, transform, imput missing values, and remove duplicates from the extracted data, ultimately obtaining a unified, highly structured, high-quality, and compatible dataset to provide reliable data support for subsequent analysis and decision-making. The format can be any format, depending on the requirements. In this step, a set of function rules are applied to the extracted data to transform it into a single standard format. It may involve the following tasks: Filtering: Loading only specific attributes into the data warehouse. Cleaning: Filling null values ​​with specific default values. Joining: Joining multiple attributes into one attribute. Splitting: Splitting a single attribute into multiple attributes. Sorting: Sorting tuples based on attributes.

[0044] Loading: Collecting large amounts of data from various sources, transforming them, and finally loading them into the data warehouse.

[0045] In general, ETL is the process of extracting data from different source systems, transforming the data, and loading the data into a data warehouse.

[0046] Stream processing is a technology that allows users to quickly query continuous data streams and detect conditions within a short period of time after receiving data. It is referred to by names such as real-time analytics, streaming analytics, complex event processing, real-time streaming analytics, and event processing. In stream processing, the input data is unrestricted and has no predetermined start or end; it simply forms a series of events that arrive at the stream processing system. Streaming is a data distribution technology where data producers write data records into an ordered data stream, from which data users can read data in the same order. It is an online data processing method that generates results by processing data streams in real time; it is generally used for processing small-scale, real-time data streams, thus allowing processing as soon as data arrives.

[0047] Batch processing: Applied to DOS and Windows systems, it is generally considered a scripting language for batch processing of an object; it is an offline data processing method that generates results by processing a batch of data; it processes data after it has been collected, so it requires a waiting period of time; batch processing can handle large amounts of data at a time, typically hundreds of thousands, millions, or tens of millions of data entries.

[0048] This invention proposes a method and apparatus for real-time statistical analysis of multi-dimensional data based on multi-table change trajectories. Based on an insurance business change trajectory table, the basic data source uses the PICC full-data Kafka platform. Using Flink streaming data processing technology, data is synchronized in real-time to an intermediate database through relevant tables of the Kafka Topic premium income indicator in the consumer payment system. Finally, batch and stream data are merged to achieve real-time statistical analysis of insurance business indicators, which are then displayed on web and mobile devices (e.g., in corresponding mobile app reports). The real-time data processing solution developed by this invention can solve the problem of slow query efficiency caused by year-on-year business growth, reduce the probability of data unloading errors, accelerate the frequency of data statistics, and ultimately improve decision-making efficiency for users.

[0049] The following is a detailed explanation.

[0050] This invention uses premium income data as an example for illustration. Real-time premium income is a summary and statistical analysis of three parts: existing data, incremental data, and updated data, as follows: Figure 1 As shown, three Flink task workflows are used to process the data, the processing result set is updated to the Redis database, and finally the "Real-time Premium Income" report in the corresponding terminal (such as a mobile APP) displays the dataset of real-time premium income summary results in the Redis database.

[0051] Figure 2 A schematic diagram illustrating the steps of a real-time statistical method for multi-dimensional data based on multi-table change trajectories provided in this embodiment of the invention may include the following steps:

[0052] S210: Retrieve existing data from the relational database Informix, perform batch processing on the existing data using ETL, and store the processed existing data in the Redis database.

[0053] In one embodiment of the present invention, existing data is obtained from the relational database Informix, and batch processing of the existing data is performed through ETL. Specifically, this includes: obtaining payment and receipt reason data and data within a statistical time range from the Informix database, converting the data into a text file through ETL processing; storing the text file in an intermediate relational database Oracle; summarizing the text file stored in the Oracle database by associating it with table information of different dimensions; periodically updating the summarized data to the Oracle result table based on a preset frequency; and loading and updating the data in the Oracle result table to the Redis database through a Flink task workflow.

[0054] In one embodiment of the present invention, data is converted into a text file through ETL processing, specifically including: data cleaning of data obtained from the Informix database, including deleting invalid data, filling missing values ​​and removing duplicate records; and format conversion of the cleaned data to obtain a text file.

[0055] Specifically, the existing data processing is responsible for statistically analyzing existing premium income data up to a preset number of days (e.g., 2 days). This processing is based on the company's traditional relational database, Informix, and uses traditional ETL processing techniques. The specific steps are as follows:

[0056] 1. Unload the data from the Sffdocdetail table in the Informix library of the payment system, which has a fixed payment reason and statistical time range, into a text file;

[0057] 2. Load the text file into the intermediate relational database Oracle;

[0058] III. Summarize information from related institutions, products, customer groups, exchange rates, and other dimensions;

[0059] IV. Update the results table in the relational database Oracle at a T+1 frequency;

[0060] Fifth, the existing summary result table data is then loaded and updated into the Redis database through the Flink task workflow.

[0061] S220: Obtain incremental data from the Kafka platform, perform stream processing on the incremental data using Flink, and store the processed incremental data in the Redis database.

[0062] In one embodiment of the present invention, incremental data is stream-processed using Flink, specifically including: consuming relevant topic data from the Kafka platform, filtering incremental data within a fixed payment / receipt reason and payment / receipt date range, storing the filtered incremental data in a relational database PostgreSQL (PG); using the incremental data as the main data stream, loading table information of different dimensions into dimension table data streams; sequentially associating the main data stream and the dimension table data streams; aggregating the associative result set according to different dimensions, merging the aggregation results belonging to the same dimension, and storing the merged dataset in a Redis database.

[0063] Incremental processing is responsible for statistically analyzing incremental premium income data within a preset number of days (e.g., within 2 days). Existing processing is based on the company's full data Kafka platform and uses Flink stream processing technology.

[0064] The specific steps are as follows:

[0065] 1. Using Flink streaming data processing technology, collect and pay related topic data from the Kafka payment platform by consuming the company's full data.

[0066] II. Filter the result set within the statistical range of fixed reasons for receipts and payments and receipt and payment dates;

[0067] 3. Store the result set in the intermediate relational database PostgreSQL;

[0068] IV. Treat the result set as the main data stream;

[0069] 5. Load relational database dimension tables such as institutions, products, and exchange rates into dimension table data streams;

[0070] 6. Sequentially associate dimension table data streams through the main data stream;

[0071] 7. Aggregate related data streams according to different dimensions;

[0072] 8. Combine the results from step 7 according to different aggregation sets;

[0073] 9. Store the final result dataset in a Redis database.

[0074] S230: Retrieve updated data from incremental data, perform batch processing on the updated data, and store the processed updated data in the Redis database.

[0075] In one embodiment of the present invention, the process of obtaining updated data from incremental data and performing batch processing on the updated data specifically includes: obtaining updated data from incremental data stored in PostgreSQL; determining historical newly added data corresponding to the updated data through primary key association; multiplying the amount in the historical newly added data by a preset value, and then merging and summarizing it with the updated data.

[0076] Specifically, batch processing is responsible for processing special update data within the incremental data. Kafka Topic source data is based on traditional database operation change trajectories, including three types of changed data: insert (I), delete (D), and update (U). Since update data involves historical data, it is processed separately. The specific steps are as follows:

[0077] 1. Identify the updated data stored in the PostgreSQL database in the third step of incremental processing;

[0078] 2. Find the historical newly added data corresponding to the updated data by using primary key association;

[0079] Third, multiply the historical newly added data amount by -1, and then merge and summarize it with the updated data.

[0080] S240: Retrieves processed existing data, incremental data, and updated data from the Redis database, summarizes and statistically analyzes them according to different dimensions, and displays the summary and statistical results in the terminal application in real time.

[0081] Through the above data processing, the existing data, incremental data, and updated data are summarized and statistically analyzed according to different dimensions, and the results are updated and displayed in real time in the reports of the mobile APP.

[0082] In one embodiment of the present invention, log monitoring is performed on the stream processing and batch processing processes to obtain log data during the stream processing and batch processing processes; multidimensional analysis is performed on the logs to determine whether there are any anomalies based on the results of the multidimensional analysis; when an anomaly is detected in the logs, the error information of the batch processing or stream processing is automatically sent to relevant personnel.

[0083] Specifically, batch processing and stream processing logs can be analyzed periodically using the following steps:

[0084] First, collect log data onto HDFS and clean the data using MapReduce or other tools, including deleting invalid data, filling in missing values, and removing duplicate records.

[0085] 2. Data Conversion: Converting data from its original format to a target format, such as date format, currency format, etc.

[0086] 3. Data merging: Merging data from multiple data sources together.

[0087] IV. Data Storage: The processed data is stored in data warehouses such as Hive.

[0088] V. Data Analysis: Use tools such as Hive to perform multidimensional analysis on the cleaned data and extract useful information.

[0089] VI. Anomaly Notification: Export the results obtained from Hive analysis and notify the operations and maintenance personnel of any anomalies via SMS and email.

[0090] In one embodiment of the present invention, a checkpointing mechanism is enabled and configured in the Flink task, and a restart strategy is configured when the task fails; possible abnormal situations are captured, and when an abnormal situation is captured, the Flink task is triggered to fail and the failed Flink task is marked; the most recent checkpoint of the marked Flink task is queried, the failed task is re-executed according to the data status of the most recent checkpoint, and recovery starts from the last saved checkpoint.

[0091] Specifically, to enable a Flink task to automatically restart and recover from the last saved checkpoint when it fails, you can follow these steps:

[0092] 1. Configure checkpoints: Enable the checkpoint mechanism in Flink tasks to ensure that the data state is periodically saved to persistent storage during task execution.

[0093] 2. Configure Restart Strategy: In Flink's configuration file (e.g., flink-conf.yaml), set the restart strategy when a task fails. Automatic task restart can be achieved by setting the `taskmanager.execution.failure-handling.strategy` parameter to `full` or `failure-rate`.

[0094] 3. Exception Handling: In Flink task code, exceptions that may occur can be caught, such as network connection failures or database connection errors. Try-catch statements can be used to catch and handle exceptions.

[0095] IV. Triggering Task Failure: When an exception is caught, task failure can be triggered by calling FailFuture or by throwing an exception. This will cause the Flink task to be marked as failed.

[0096] 5. Task Recovery: When a Flink task fails, Flink will attempt to recover the task from the most recent checkpoint. Flink will re-execute the failed task based on the data state of the checkpoint and start recovery from the last saved checkpoint.

[0097] 6. Continuous Monitoring: During the execution of a Flink task, it is necessary to continuously monitor the task's execution status and error logs. If an exception or task failure is detected, ensure that the task failure capture and handling logic is triggered in a timely manner.

[0098] The method of this invention can achieve the following:

[0099] 1. Based on the real-time trajectory changes of traditional database business data, it supports real-time data statistics for deletion and updates;

[0100] 2. Based on the data statistics characteristics of the company's payment system, the system payment time is used as the statistical interval identifier. Data older than 2 days is statistically processed by batch processing, data within 2 days is statistically processed by stream processing, and special updated data is updated by separate batch processing on a timed basis. Ultimately, this achieves seamless matching of batch data and stream data, realizing real-time data statistics integrating batch and stream processing.

[0101] 3. Supports automatic alarm for anomalies. By periodically analyzing batch and stream processing logs through monitoring scripts, when database error information or Java exception error information is found, the error information of batch processing or stream processing is automatically notified to the operation and maintenance personnel via SMS and email.

[0102] 4. When a Flink task program reports an error due to network or database abnormalities, it supports automatic restart of the Flink task based on the last saved checkpoint.

[0103] The above describes a real-time statistical method for multi-dimensional data based on the change trajectory of multiple tables, as provided in an embodiment of the present invention. Based on the same inventive concept, this invention also provides a corresponding real-time statistical device for multi-dimensional data based on the change trajectory of multiple tables, such as... Figure 3 As shown.

[0104] The existing data processing module 302 is used to obtain existing data from the relational database Informix, perform batch processing on the existing data through ETL, and store the processed existing data in the Redis database.

[0105] The incremental data processing module 304 is used to obtain incremental data from the Kafka platform, perform stream processing on the incremental data through Flink, and store the processed incremental data in the Redis database.

[0106] The update data processing module 306 is used to obtain the update data from the incremental data, perform batch processing on the update data, and store the processed update data in the Redis database;

[0107] The summary statistics module 308 is used to retrieve processed existing data, incremental data, and updated data from the Redis database, and perform summary statistics according to different dimensions, and update and display the summary statistics results in the terminal application in real time.

[0108] This invention also provides a corresponding real-time statistical device for multi-dimensional data based on multi-table change trajectories, such as... Figure 4 As shown, it includes:

[0109] The system includes at least one processor 402, a communication interface 404, a memory 406, and a communication bus 408; wherein the processor 402, the communication interface 404, and the memory 406 communicate with each other through the communication bus 408; the processor 402 can call the logical instructions stored in the memory 406 to cause at least one processor 402 to execute the steps of the above embodiments.

[0110] Based on the same idea, some embodiments of the present invention also provide media corresponding to the above methods.

[0111] Some embodiments of the present invention provide a storage medium storing computer-executable instructions, which are executed by a processor to implement the steps of the embodiments described above.

[0112] The various embodiments in this invention are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and medium embodiments are relatively simple in description because they are fundamentally similar to the method embodiments; relevant parts can be referred to the descriptions in the method embodiments.

[0113] The devices, media, and methods provided in the embodiments of the present invention are one-to-one correspondences. Therefore, the devices and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0114] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process method or product that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process method or product. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process method or product that includes that element.

[0115] The above are merely embodiments of the present invention and are not intended to limit the invention. Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A real-time statistical method for multi-dimensional data based on multi-table change trajectories, characterized in that, include: The process involves retrieving existing data from the Informix relational database, performing batch processing on the existing data using ETL, and storing the processed existing data in a Redis database. Specifically, this includes: retrieving payment / receipt reason data and data within a statistical time range from the Informix database; converting the data into a text file using ETL; storing the text file in an intermediate relational database (Oracle); summarizing the text file stored in the Oracle database by associating it with table information from different dimensions, including organization, product, customer group, and exchange rate; periodically updating the summarized data in the Oracle result table based on a preset frequency; and loading and updating the data in the Oracle result table into the Redis database using a Flink task workflow. Incremental data is obtained from the Kafka platform, stream-processed using Flink, and then stored in the Redis database. Obtain the updated data from the incremental data, perform batch processing on the updated data, and store the processed updated data in the Redis database; The system retrieves processed existing data, incremental data, and updated data from the Redis database, summarizes and statistically analyzes them according to different dimensions, and displays the summarized statistical results in the terminal application in real time.

2. The method according to claim 1, characterized in that, The incremental data stream processing via Flink specifically includes: Consume relevant topic data from the Kafka platform, filter out incremental data within a fixed payment / receipt reason and payment / receipt date range, and store the filtered incremental data in a relational database PostgreSQL; The incremental data is used as the main data stream, and table information of different dimensions is loaded into dimension table data streams. The main data stream and the dimension table data stream are sequentially associated; The associated result set is aggregated according to different dimensions, the aggregated results belonging to the same dimension are merged, and the merged dataset is stored in the Redis database.

3. The method according to claim 2, characterized in that, The step of obtaining updated data from the incremental data and performing batch processing on the updated data specifically includes: Retrieve updated data from incremental data stored in PostgreSQL; The historical newly added data corresponding to the updated data is determined by primary key association; The amount in the historical newly added data is multiplied by a preset value and then merged and summarized with the updated data.

4. The method according to claim 1, characterized in that, The step of converting the data into a text file through ETL processing specifically includes: Data cleaning is performed on the data obtained from the Informix database, including deleting invalid data, filling in missing values, and removing duplicate records; The cleaned data is then converted to a text file.

5. The method according to claim 1, characterized in that, The method further includes: Perform log monitoring on the stream processing and batch processing processes to obtain log data during the stream processing and batch processing processes; Perform multidimensional analysis on the logs and determine whether there are any anomalies based on the results of the multidimensional analysis; When log anomalies are detected, error messages from batch or stream processing will be automatically sent to the relevant personnel.

6. The method according to claim 5, characterized in that, The method further includes: Enable and configure checkpointing in Flink tasks, and configure restart strategies when tasks fail. Capture potential exceptions. When an exception is captured, trigger a Flink task failure and mark the failed Flink task. Query the most recent checkpoint of the marked Flink task, re-execute the failed task based on the data state of the most recent checkpoint, and resume from the last saved checkpoint.

7. A real-time statistical device for multi-dimensional data based on multi-table change trajectories, characterized in that, include: The existing data processing module is used to obtain existing data from the relational database Informix, perform batch processing on the existing data through ETL, and store the processed existing data in the Redis database. Specifically, it includes: obtaining payment and receipt reason data and data within a statistical time range from the Informix database; converting the data into a text file through ETL; storing the text file in an intermediate relational database Oracle; associating the text file stored in the Oracle database with table information of different dimensions, including organization, product, customer group, and exchange rate; updating the summarized data to the Oracle result table at a preset frequency; and loading and updating the data in the Oracle result table to the Redis database through a Flink task workflow. The incremental data processing module is used to obtain incremental data from the Kafka platform, perform stream processing on the incremental data through Flink, and store the processed incremental data in the Redis database. An update data processing module is used to obtain the updated data from the incremental data, perform batch processing on the updated data, and store the processed updated data in a Redis database. The summary and statistics module is used to retrieve the processed existing data, incremental data, and updated data from the Redis database, and perform summary statistics according to different dimensions, and update and display the summary and statistics results in the terminal application in real time.

8. A real-time statistical device for multi-dimensional data based on multi-table change trajectories, characterized in that, include: At least one processor; as well as, The memory is communicatively connected to the at least one processor via a bus; wherein, The memory stores instructions executable by the at least one processor, which are executed to implement the method as described in any one of claims 1-6.

9. A non-volatile storage medium storing computer-executable instructions, characterized in that, The computer-executable instructions are executed by a processor to implement the method as described in any one of claims 1-6.

Images