Visual data early warning system and method

Abstract

The invention provides a visual data early warning method and system. The method comprises the steps of S1, data source configuration and access, S2, data preprocessing and cleaning, S3, real-time monitoring and acquisition, S4, early warning rule setting, S5, data analysis and prediction, S6, early warning triggering and notification and S7, visual display and interaction. According to the invention, through the functions of real-time monitoring, prediction analysis, multi-dimensional display, rapid response, decision support and the like, accurate early warning of various risk indexes of an enterprise is realized; the method comprises the steps of determining a corresponding service rule code set by acquiring service data of an application end in real time, calling a corresponding early warning rule operation model according to a set early warning trigger mode, calculating an early warning value according to an early warning rule, and sending the early warning value to a receiving end in a graphical display form when the early warning value meets an early warning trigger condition. And efficient service data early warning trigger analysis and message early warning can be realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a visualized data early warning system and method. Background Art

[0002] At present, many enterprises have realized the regular reporting of regulatory data. Regulatory agencies have strict requirements on the quality of regulatory reporting data. Before each reporting, the reported data must be checked for data quality and relevant data repair operations must be completed. The previous detection method was to perform manual data detection through SQL statements. Data collection was not timely, and the lag in discovering erroneous data led to differences in the quality of the reported data. Summary of the invention

[0003] The purpose of the present invention is to solve the shortcomings existing in the prior art, and to propose a visualized data early warning system and method, which determines the corresponding business rule coding set by acquiring the business data of the application end in real time, calls the corresponding early warning rule operation model according to the set early warning trigger mode, calculates the early warning value according to the early warning rule, and sends it to the receiving end in a graphical display form when the early warning value meets the early warning trigger condition, so as to realize efficient business data early warning trigger analysis and message early warning.

[0004] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a visual data early warning system, which includes a data source configuration module, an administrator login module, a data preprocessing and cleaning module, a real-time data monitoring and acquisition module, an early warning rule setting module, a data analysis and prediction module, an early warning trigger and notification module, and a visual display and interaction module; the administrator login module is electrically connected to the remaining modules; the data source configuration module is electrically connected to the real-time data monitoring and acquisition module; the real-time data monitoring and acquisition module is electrically connected to the data preprocessing and cleaning module; the data analysis and prediction module is electrically connected to the data preprocessing and cleaning module, the early warning rule setting module, and the early warning trigger and notification module respectively; the early warning trigger and notification module is also electrically connected to the visual display and interaction module.

[0005] A visual data early warning method, the method is specifically as follows: S1: Data source configuration and access: The administrator configures the data source through the system interface, specifies the database type and its connection information, including database IP, port, user name, and password; the early warning system can then establish a connection with the database of the business system to obtain real-time data; The query terminal interacts with the data platform through wired or wireless communication; S2: Data preprocessing and cleaning: Re-examine and verify the data obtained in step S1 to detect and correct errors, omissions and inconsistencies in the data; This includes removing duplicate data to ensure that the data set does not contain duplicate records or rows; handling missing values, estimating, deleting or replacing missing data to improve data integrity; correcting incorrectly formatted data and converting data into the correct format, such as date format or numerical format; S3: Real-time monitoring and collection: Schedule data extraction tasks through the task scheduling platform, extract data from the data source into the system in real time, set timestamps on the source end, extract data with timestamp changes into the system in seconds to ensure the real-time and accuracy of the data, monitor data extraction in real time, and prompt the administrator for tasks that fail to extract data in time to make adjustments and re-extract data; S4: Warning rule setting: According to the actual demand for data accuracy of the data reported in step S3, the warning rules are set in reverse order, such as setting a threshold of a certain indicator, and triggering a warning when the indicator exceeds or falls below the threshold, such as setting a reasonable range of a certain indicator, and triggering a warning when the indicator falls below or exceeds the range; S5: Data Analysis and Prediction: Perform data analysis and prediction. On the task scheduling platform, execute the detection script and conduct targeted analysis of the data according to the set warning rules. If there is data that does not meet the rules or exceeds the scope of the rules, save the subject, organization, error type and other information of the data, and trigger the warning mechanism. Set up the task scheduling platform, and the task scheduling time interval for each rule shall not exceed 10 seconds to ensure that the problem data is triggered in time; S6: Warning trigger and notification: The error data detected by the early warning rules can be visualized and automatically notified to the user through system notification, email notification, and SMS notification. S7: Visual display and interaction: Provides a variety of visual charts to display abnormal points of risk indicators for display on terminal devices, which can be computers, mobile phones, and Pads. It supports access to user-defined chart styles and layouts, making it convenient for users to adjust according to actual needs.

[0006] Preferably, in step S1, the system supports access to multiple data sources, including Excel files and various types of databases, to ensure the comprehensiveness and real-time nature of the data.

[0007] Preferably, in step S4, the core purpose of the early warning mechanism is to ensure that the data is accurate and meets predetermined business or compliance requirements by real-time monitoring of various indicators; according to actual needs, different rules can be set for detection and alarm, including threshold setting warning, reasonable range warning and multi-condition combination warning.

[0008] Preferably, in step S5, data analysis and detection include: Real-time detection: On the task scheduling platform, each time a task is run, the detection script will analyze the data in real time. The script will identify the data that does not meet the rules by applying the warning rules, mark them, and record the relevant information. Data subject and agency identification: During the detection process, the source and ownership of the data need to be clear; the subject of the data, the agency to which it belongs, and the type of error need to be recorded.

[0009] Error type classification: The detection script also needs to classify unqualified data according to the preset error type to facilitate subsequent processing and analysis.

[0010] Compared with the prior art, the present invention has the following beneficial effects: through real-time monitoring and predictive analysis, the present invention can timely discover and predict risks and improve the accuracy of early warning. Secondly, it is also equipped with an automatic trigger alarm mechanism to ensure that relevant personnel can respond to risks quickly. It also provides comprehensive data analysis and visualization functions to provide strong support for decision makers; and through the visualization of risk indicators, managers can more intuitively understand the risk status and improve management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 The figure is a schematic diagram of the overall process; Figure 2 This is a schematic diagram of the module connection. DETAILED DESCRIPTION

[0012] In order to provide a further understanding of the purpose, structure, features, and functions of the present invention, the following detailed description is given in conjunction with the embodiments.

[0013] Please refer to Figure 1 as well as Figure 2The present invention provides a visualized data early warning system, which includes a data source configuration module, an administrator login module, a data preprocessing and cleaning module, a real-time data monitoring and acquisition module, an early warning rule setting module, a data analysis and prediction module, an early warning trigger and notification module, and a visualization and interaction module; the administrator login module is electrically connected to the remaining modules; the data source configuration module is electrically connected to the real-time data monitoring and acquisition module; the real-time data monitoring and acquisition module is electrically connected to the data preprocessing and cleaning module; the data analysis and prediction module is electrically connected to the data preprocessing and cleaning module, the early warning rule setting module, and the early warning trigger and notification module respectively; the early warning trigger and notification module is also electrically connected to the visualization and interaction module.

[0014] The data early warning system achieves efficient data flow and collaborative work through the electrical signal connection between each module. The administrator login module ensures system security and authority management, and the connection between the data source configuration module and the real-time data monitoring and acquisition module ensures the accuracy and real-time nature of the data source. The linkage between the real-time data monitoring and the data preprocessing and cleaning module helps to effectively clean and preprocess the data after collection, thereby improving the accuracy of subsequent analysis. The combination of the data analysis and prediction module with the early warning rule setting module and the early warning trigger and notification module can perform intelligent analysis of the data in real time according to the set rules, trigger early warnings, and provide timely feedback to the administrator through the notification module. In addition, the linkage between the early warning trigger and notification module and the visual display and interaction module enables the early warning information to be displayed in an intuitive and easy-to-understand way, enhancing the operability and response efficiency of the system. The overall design promotes the fluency, accuracy and timeliness of data processing, and effectively improves the flexibility and accuracy of early warning response.

[0015] A visual data early warning method, the method is specifically as follows: S1: Data source configuration and access: The administrator configures the data source through the system interface, specifies the database type and its connection information, including database IP, port, user name, and password; the early warning system can then establish a connection with the database of the business system to obtain real-time data; The query terminal interacts with the data platform through wired or wireless communication.

[0016] The data early warning system is configured by the system administrator to collect data from the source. The query terminal can communicate with the financial data platform through wired or wireless communication. The system can be implemented using a web page in B / S mode or a server in C / S mode. The specific implementation method does not constitute a limitation of the present invention. The data source configured here is the Gauss DB database of the business system.

[0017] When the administrator logs in, he carries a user ID, where the user ID refers to an ID that has the authority to configure the data source. In other words, the data warning platform can be a network platform shared by multiple users, and the administrator user can determine the identity of the user by using a pre-registered user name and password when logging into the platform.

[0018] The system supports access to multiple data sources, including Excel files and various types of databases, to ensure the comprehensiveness and real-time nature of the data; the data source can be flexibly configured, allowing users to add or delete data sources based on actual needs. During configuration, it is necessary to specify the storage path of the Excel file or the access IP address and port of the server where the database is located, and ensure that it is accessible.

[0019] S2: Data preprocessing and cleaning: Re-examine and verify the data obtained in step S1 to detect and correct errors, omissions and inconsistencies in the data; It includes removing duplicate data to ensure that the data set does not contain duplicate records or rows; handling missing values, estimating, deleting or replacing missing data to improve data integrity; correcting malformed data and converting data into the correct format, such as date format or numeric format.

[0020] For example, in the preprocessing of regulatory reporting data, due to historical reasons, the customer's gender is not a value of "male" or "female". For such data, a value will be assigned based on the gender check digit of the ID number. For the end date of the policy, a value will be assigned based on the effective date plus the insurance period minus one day. For the customer's contact address, if the first three sections of "province", "city" and "county" are empty, this part of the data will be considered and not included in the detection scope.

[0021] S3: Real-time monitoring and collection: Through the task scheduling platform, data extraction tasks are scheduled to extract data from the data source into the system in real time. The timestamp is set at the source. For data with changing timestamps, it is extracted into the system in seconds to ensure the real-time and accuracy of the data. The data extraction status is monitored in real time. For tasks that fail to extract, the system prompts the administrator to make timely adjustments and re-extract data.

[0022] The task scheduling platform achieves real-time extraction of data sources by setting timestamps, ensuring that data changes can be updated to the system at the second level, thereby maintaining the real-time and accuracy of data. The real-time monitoring mechanism can promptly capture extraction failures and help administrators quickly adjust and re-extract data through system prompts to ensure data integrity and stability.

[0023] S4: Warning rule setting: According to the actual demand for data accuracy of the data reported in step S3, the warning rules are set in reverse order, such as setting a threshold for a certain indicator, triggering a warning when the indicator exceeds or falls below the threshold, such as setting a reasonable range for a certain indicator, triggering a warning when the indicator falls below or exceeds the range.

[0024] According to the data reporting requirements of step S3 supervision, the accuracy of the data is crucial, so it is necessary to set early warning rules to ensure the validity and reliability of the data. By setting indicator thresholds or reasonable ranges, data fluctuations can be monitored in real time and abnormal situations can be discovered in a timely manner. For example, when an indicator exceeds or falls below a predetermined threshold, the system will automatically trigger an early warning to prompt relevant personnel to review; similarly, when an indicator deviates from a preset reasonable range, an early warning mechanism will also be triggered. These early warning rules help to identify potential problems in a timely manner, ensure the accuracy and compliance of data, and thus improve the efficiency and accuracy of supervision.

[0025] The core purpose of the early warning mechanism is to ensure that the data is accurate and meets the predetermined business or compliance requirements by monitoring various indicators in real time; according to actual needs, different rules can be set for detection and alarm, including threshold setting warning, reasonable range warning and multi-condition combination warning.

[0026] S5: Data Analysis and Prediction: Perform data analysis and prediction. On the task scheduling platform, execute the detection script and conduct targeted analysis of the data according to the set warning rules. If there is data that does not meet the rules or exceeds the scope of the rules, save the subject, organization, error type and other information of the data, and trigger the warning mechanism. Set up the task scheduling platform, and the task scheduling time interval for each rule shall not exceed 10 seconds to ensure that the problem data is triggered in time.

[0027] Data analysis and prediction are performed on the task scheduling platform, and the data is monitored in real time and analyzed in a targeted manner by running detection scripts regularly. According to the set early warning rules, the system will promptly identify data that does not meet the standards or exceeds the scope of the rules, and save relevant information, such as the subject to which the data belongs, the affiliated institution, and the type of error, for subsequent analysis and processing. At the same time, the task scheduling time interval does not exceed 10 seconds to ensure that the execution frequency of each rule is high enough, so that the early warning mechanism can be quickly triggered when the data is abnormal, ensuring data quality and timely response of the system.

[0028] Data analysis and detection include: Real-time detection: On the task scheduling platform, each time a task is run, the detection script will analyze the data in real time. The script will identify the data that does not meet the rules by applying the warning rules, mark them, and record the relevant information; Data subject and agency identification: During the detection process, the source and ownership of the data need to be clear; the subject of the data, the agency to which it belongs, and the type of error need to be recorded.

[0029] Error type classification: The detection script also needs to classify unqualified data according to the preset error type to facilitate subsequent processing and analysis.

[0030] S6: Warning trigger and notification: The error data detected by the early warning rules can be visualized and automatically notified to the user through system notification, email notification, and SMS notification. S7: Visual display and interaction: Provides a variety of visual charts to display abnormal points of risk indicators for display on terminal devices, which can be computers, mobile phones, and Pads. It supports access to user-defined chart styles and layouts, making it convenient for users to adjust according to actual needs.

[0031] Users can customize the style and layout of the chart, such as adjusting the chart color, font size, title, etc. Supports chart export function, which is convenient for users to save charts locally or share them with others. Provides data analysis and visualization functions to provide decision makers with comprehensive and accurate data support. Chart type selection: The system provides a variety of visualization chart types, such as maps, bar charts, line charts, pie charts, etc. Users can choose the appropriate chart type to display data according to actual needs. Chart style and layout adjustment: Users can customize the style and layout of the chart, such as adjusting the chart color, font size, title, etc. Supports chart dragging and zooming functions, which is convenient for users to adjust according to actual needs. Supports chart export function, which is convenient for users to save charts locally or share them with others.

[0032] The present invention has been described by the above-mentioned relevant embodiments, however, the above-mentioned embodiments are only examples for implementing the present invention. It must be pointed out that the disclosed embodiments do not limit the scope of the present invention. On the contrary, changes and modifications made without departing from the spirit and scope of the present invention are all within the scope of patent protection of the present invention.

Claims

1. A visual data early warning system, characterized by: The data early warning system includes a data source configuration module, an administrator login module, a data preprocessing and cleaning module, a real-time data monitoring and acquisition module, an early warning rule setting module, a data analysis and prediction module, an early warning trigger and notification module, and a visual display and interaction module; the administrator login module is electrically connected to the remaining modules; the data source configuration module is electrically connected to the real-time data monitoring and acquisition module; the real-time data monitoring and acquisition module is electrically connected to the data preprocessing and cleaning module; the data analysis and prediction module is electrically connected to the data preprocessing and cleaning module, the early warning rule setting module, and the early warning trigger and notification module respectively; the early warning trigger and notification module is also electrically connected to the visual display and interaction module.

2. An early warning method of a visual data early warning system as claimed in claim 1, characterized in that: The method is specifically as follows: S1: Data source configuration and access: The administrator configures the data source through the system interface, specifies the database type and its connection information, including database IP, port, user name, and password; the early warning system can then establish a connection with the database of the business system to obtain real-time data; The query terminal interacts with the data platform through wired or wireless communication; S2: Data preprocessing and cleaning: Re-examine and verify the data obtained in step S1 to detect and correct errors, omissions and inconsistencies in the data; This includes removing duplicate data to ensure that the data set does not contain duplicate records or rows; handling missing values, estimating, deleting or replacing missing data to improve data integrity; correcting incorrectly formatted data and converting data into the correct format, such as date format or numerical format; S3: Real-time monitoring and collection: Schedule data extraction tasks through the task scheduling platform, extract data from the data source into the system in real time, set timestamps on the source end, extract data with timestamp changes into the system in seconds to ensure the real-time and accuracy of the data, monitor data extraction in real time, and prompt the administrator for tasks that fail to extract data in time to make adjustments and re-extract data; S4: Warning rule setting: According to the actual demand for data accuracy of the data reported in step S3, the warning rules are set in reverse order, such as setting a threshold of a certain indicator, and triggering a warning when the indicator exceeds or falls below the threshold, such as setting a reasonable range of a certain indicator, and triggering a warning when the indicator falls below or exceeds the range; S5: Data Analysis and Prediction: Perform data analysis and prediction. On the task scheduling platform, execute the detection script and conduct targeted analysis of the data according to the set warning rules. If there is data that does not meet the rules or exceeds the scope of the rules, save the subject, organization, error type and other information of the data, and trigger the warning mechanism. Set up the task scheduling platform, and the task scheduling time interval for each rule shall not exceed 10 seconds to ensure that the problem data is triggered in time; S6: Warning trigger and notification: The error data detected by the early warning rules can be visualized and automatically notified to the user through system notification, email notification, and SMS notification. S7: Visual display and interaction: Provides a variety of visual charts to display abnormal points of risk indicators for display on terminal devices, which can be computers, mobile phones, and Pads. It supports access to user-defined chart styles and layouts, making it convenient for users to adjust according to actual needs.

3. A visual data early warning method as claimed in claim 2, characterized in that: In step S1, the system supports access to multiple data sources, including Excel files and various types of databases, to ensure the comprehensiveness and real-time nature of the data.

4. A visual data early warning method as claimed in claim 2, characterized in that: In step S4, the core purpose of the early warning mechanism is to ensure that the data is accurate and meets the predetermined business or compliance requirements by real-time monitoring of various indicators; According to actual needs, different rules can be set for detection and alarm, including threshold setting warning, reasonable range warning and multi-condition combination warning.

5. A visual data early warning method as claimed in claim 2, characterized in that: In step S5, data analysis and detection include: Real-time detection: On the task scheduling platform, each time a task is run, the detection script will analyze the data in real time. The script will identify the data that does not meet the rules by applying the warning rules, mark them, and record the relevant information. Data subject and agency identification: During the detection process, the source and ownership of the data need to be clear; the subject, agency and error type of the data need to be recorded; Error type classification: The detection script also needs to classify unqualified data according to the preset error type to facilitate subsequent processing and analysis.

Images