Quality analysis and early warning method and device for service submission data, equipment and medium

Abstract

The invention relates to the technical field of big data and the field of financial science and technology, and discloses a quality analysis and early warning method for service submission data, which extracts specified information by receiving and processing submission items submitted by a service submission end. Based on the information, SQL statements are generated, execution of the SQL statements is triggered according to a timed task, and service data of corresponding batches are extracted from a database. And carrying out quality analysis on the extracted data, analyzing dimensions including integrity, validity, uniqueness, accuracy and consistency, and generating a quality analysis result. And setting a threshold value for triggering early warning according to the historical data analysis result. And when the quality analysis result is lower than a preset threshold value, triggering early warning and notifying related personnel, wherein the early warning information comprises the data quality dimension, the evaluation result and the early warning level. According to the invention, through automatic processing and monitoring, the accuracy and integrity of submitted data are improved, the risk of wrong reporting and missing reporting is effectively reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the fields of big data technology and financial technology, and in particular to a method, device, equipment and storage medium for quality analysis and early warning of business reporting data. Background Art

[0002] In the financial and insurance industries, regulatory reporting requirements are becoming increasingly complex, and the number of reporting items is increasing. Existing regulatory reporting systems usually rely on a single reporting data processing architecture, which is often composed of multiple independent modules with limited collaboration capabilities between modules. Existing technologies face many challenges and deficiencies in processing business reporting data.

[0003] Databases, data tables and other information are distributed in different systems, and the data reported by each department or institution is relatively independent, lacking a unified management and coordination mechanism. This data management method easily leads to problems such as data redundancy and repeated development, making it difficult to form a centralized and systematic data management process.

[0004] The existing reporting system mainly relies on simple rule verification or data matching functions in data quality analysis, and is unable to deeply analyze key dimensions such as data integrity, validity, uniqueness, accuracy and consistency. Due to the single analysis method, it is difficult for the system to fully identify problems in the data and to correct and optimize the data in a timely manner, which brings potential risks to the reporting task.

[0005] Existing regulatory reporting technologies usually rely on manual operations or single automated tasks in data processing and reporting task execution, and lack a comprehensive automated management mechanism. Especially when processing complex reporting data, manual operations and single-task automation are not enough to cope with the increasing reporting requirements, resulting in low reporting efficiency and increased error rates.

[0006] The current reporting system is still passive in terms of data anomaly monitoring and early warning mechanisms. Although some systems can detect anomalies in reported data, they are usually discovered after the fact and cannot take immediate action when errors occur in reported data. The lack of real-time monitoring and early warning functions makes the reporting task face a higher risk, especially when the data quality does not meet the standards, it is difficult to trigger early warnings and notify relevant personnel in a timely manner. Summary of the invention

[0007] The main purpose of the present invention is to provide a method, device, equipment and storage medium for quality analysis and early warning of business reporting data, aiming to solve the technical problem that the existing technology is unable to perform systematic quality analysis and real-time early warning on business reporting data, resulting in a high risk of data misreporting and omission.

[0008] To achieve the above object, the present invention provides a quality analysis and early warning method for business reporting data, comprising:

[0009] Obtain and process the business reporting items submitted by the business reporting terminal, wherein the business reporting items specify the data source, database, data table, data field and scheduled task;

[0010] Generate SQL statements according to the data source, database, data table and data field specified in the business report item;

[0011] By means of the scheduled task, the SQL statement is executed regularly to extract the business submission data of the corresponding submission batch from the specified data source and database;

[0012] Based on the business reporting data, analyze the integrity, validity, uniqueness, accuracy and consistency of the data, and generate quality analysis results for the corresponding business reporting items;

[0013] Set thresholds for early warning based on historical data analysis results of business reporting items;

[0014] When the quality analysis result of the business report item is lower than the set threshold, an early warning is triggered, and the early warning information is sent to the business report terminal.

[0015] Furthermore, in order to achieve the above-mentioned purpose, the present invention provides a quality analysis and early warning device for business reporting data, comprising:

[0016] The business report processing module is used to obtain and process the business report items submitted by the business report terminal, wherein the business report items specify the data source, database, data table, data field and scheduled task;

[0017] An SQL generation module is used to generate SQL statements according to the data source, database, data table and data field specified in the business report item;

[0018] A data extraction module is used to periodically execute the SQL statement through the scheduled task to extract the business submission data of the corresponding submission batch from the specified data source and database;

[0019] A data quality analysis module, used to analyze the integrity, validity, uniqueness, accuracy and consistency of the data based on the business report data, and generate quality analysis results for the corresponding business report items;

[0020] The warning threshold setting module is used to set the threshold for warning according to the historical data analysis results of the business report items;

[0021] The early warning notification module is used to trigger an early warning when the quality analysis result of the business report item is lower than a set threshold, and send the early warning information to the business report terminal.

[0022] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer device, which includes a memory, a processor, and a quality analysis and early warning program for business reporting data stored in the memory and executable on the processor, and the quality analysis and early warning program for business reporting data, when executed by the processor, implements the steps of the quality analysis and early warning method for business reporting data as described above.

[0023] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a quality analysis and early warning program for business reporting data is stored. When the quality analysis and early warning program for business reporting data is executed by a processor, the steps of the quality analysis and early warning method for business reporting data as described above are implemented.

[0024] Beneficial effects: The present invention relates to the fields of big data technology and financial technology, and discloses a quality analysis and early warning method for business reporting data. By receiving and processing the reporting items submitted by the business reporting terminal, the specified data source, database, data table, data field and scheduled task information are extracted. Based on this information, an SQL statement is generated, and the execution of the SQL statement is triggered according to the scheduled task to extract the corresponding batch of business data from the database. The extracted data is subjected to quality analysis, and the analysis dimensions include integrity, validity, uniqueness, accuracy and consistency to generate quality analysis results. According to the historical data analysis results, a threshold for triggering an early warning is set. When the quality analysis result is lower than the preset threshold, an early warning is triggered and relevant personnel are notified. The early warning information includes data quality dimensions, evaluation results and early warning levels. The present invention improves the accuracy and integrity of the reported data through automated processing and monitoring, effectively reduces the risk of misreporting and underreporting, improves data processing efficiency, enhances the ability to identify and handle data quality problems in the early stage, and ensures the standardization and compliance of the reporting tasks of the financial insurance industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0026] Figure 1 A schematic diagram of an application environment of a quality analysis and early warning method for business reporting data in an embodiment of the present invention;

[0027] Figure 2 A flow chart of an embodiment of a quality analysis and early warning method for business reporting data of the present invention;

[0028] Figure 3A schematic diagram of functional modules of a preferred embodiment of the quality analysis and early warning device for business reporting data of the present invention;

[0029] Figure 4 A schematic diagram of the structure of a computer device in one embodiment of the present invention;

[0030] Figure 5 FIG. 4 is another schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0031] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0032] The quality analysis and early warning method of business reporting data provided by the embodiment of the present invention can be applied in Figure 1 In an application environment, the user terminal communicates with the server terminal through a network. The server terminal can receive and process the reporting items submitted by the business reporting terminal through the user terminal, and extract the specified data source, database, data table, data field and scheduled task information. Based on this information, an SQL statement is generated, and the execution of the SQL statement is triggered according to the scheduled task to extract the corresponding batch of business data from the database. The extracted data is subjected to quality analysis, and the analysis dimensions include integrity, validity, uniqueness, accuracy and consistency, and a quality analysis result is generated. According to the historical data analysis results, a threshold for triggering an early warning is set. When the quality analysis result is lower than the preset threshold, an early warning is triggered and relevant personnel are notified. The early warning information includes data quality dimensions, evaluation results and early warning levels. The present invention improves the accuracy and integrity of the reported data through automated processing and monitoring, effectively reduces the risk of misreporting and omissions, improves data processing efficiency, enhances the ability to identify and handle data quality problems in the early stage, and ensures the standardization and compliance of the reporting tasks of the financial insurance industry. Among them, the user terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server terminal can be implemented with an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific embodiments.

[0033] See also Figure 2 , Figure 2 This is a flow chart of an embodiment of a quality analysis and early warning method for business reporting data provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] like Figure 2 As shown, the quality analysis and early warning method of business reporting data proposed by the present invention includes the following steps:

[0035] S10, obtaining and processing the business reporting items submitted by the business reporting terminal, wherein the business reporting items specify the data source, database, data table, data field and scheduled task;

[0036] In this embodiment, the business reporting end refers to the system, application or organization that submits business data for supervision or internal reporting. It is responsible for aggregating business data from the business source and submitting it to the data processing system or superior organization to ensure that the data is submitted in a timely manner in accordance with the prescribed format and requirements. Generally, the business reporting end can be a different system platform or entity organization, which is responsible for processing and submitting business-related data to the central database or supervision system.

[0037] Among them, the headquarters business reporting terminal includes:

[0038] Financial system reporting end: The headquarters' financial department submits financial data to the regulatory department or internal audit department through a dedicated financial reporting system, including monthly reports, quarterly reports, annual reports, etc. For example, the headquarters' financial system may regularly extract information from the financial data of each subsidiary and submit it in a summary.

[0039] Operation processing and reporting end: The headquarters' operation processing system regularly reports KPIs, strategic execution and other data related to the company's operations. This system is mainly responsible for the reporting data required by senior management, including the company's overall operational health and efficiency indicators.

[0040] Compliance reporting: The compliance department is responsible for submitting reports and data related to risk management and legal compliance to regulatory agencies, such as the compliance inspection data of the State Financial Supervision and Administration Bureau, through the internal compliance reporting system.

[0041] In addition, it also includes the business reporting end of local institutions:

[0042] Financial reporting of local branches: Local branches have independent financial systems to summarize financial information within the region, and then regularly report the data to the headquarters as the basic data for financial reporting by the headquarters.

[0043] Sales reporting end of local branches: The sales department uses the sales processing system to submit quarterly performance reports, including customer information, sales, product sales, etc. The data is aggregated to local agencies and then submitted to the headquarters system through a unified interface.

[0044] Local branch customer service reporting terminal: The local branch’s customer service system submits data on customer complaints, service request processing, insurance claims, etc., and reports to the headquarters within a specified period to summarize the service quality data across the country.

[0045] Receive submission items from the business submission end. Submission items contain multiple data sets related to the business. The acquisition process is usually implemented through API interfaces, system integration interfaces or message queues, which can ensure that the data from the business submission end can be stably and securely transmitted to the system.

[0046] You can use HTTP API requests to receive data. You can also use message queues such as Kafka and RabbitMQ to asynchronously receive large batches of data. Use two-way authentication to ensure the security of data transmission.

[0047] Processing involves classifying, storing and processing the received business submissions. The system needs to perform preliminary processing on each business submission, including parsing and verifying data sources, databases, data tables, data fields and scheduled tasks to ensure data integrity and availability.

[0048] Data storage can use relational databases (such as MySQL, PostgreSQL) or NoSQL databases (such as MongoDB) to save business reporting items. Data parsing can be performed in JSON or XML format to extract data sources, databases and other elements in the reporting items, and verify and check them according to the configuration. For data verification, regular expressions and data type matching can be used to ensure the accuracy of field values.

[0049] The data source specifies the actual location where the reported data is stored, which may be an internal database, an external API, or a third-party data warehouse. The system needs to be able to correctly connect to and read the specified data source for subsequent data processing. The data source can be a SQL database (such as MySQL, Oracle) or a NoSQL database (such as MongoDB, Redis). The connection method can use the JDBC connection pool to ensure efficient connection under multi-concurrency and support the configuration of multiple data sources.

[0050] The database designation defines the specific database where the reporting item data is located. The system reads or writes related data tables by connecting to the specified database. Use the database connection pool to configure different database instances. Dynamically select different database instances to connect to based on the configuration in the business reporting item.

[0051] The data table specifies the table structure for storing business report data. The system reads data or performs data quality analysis based on the specified data table in the report item. Query or insert operations on the specified data table are performed through SQL statements or ORM frameworks (such as Hibernate). Metadata analysis of the data table can obtain table structure information through the metadata interface provided by the database system.

[0052] Data fields represent specific columns or attributes in the submitted data. The system needs to ensure that the data fields in the submitted items are consistent with the fields in the actual database table to perform effective read and write operations. Obtain the field information of the data table through database query and compare it with the fields in the submitted items to ensure field consistency. The field mapping function can be implemented to solve the problem of inconsistent field naming.

[0053] Scheduled tasks control the scheduling and execution frequency of data processing. The report items may contain information such as the execution time and frequency of the task. The system needs to perform data extraction, data analysis and other operations on time based on this information. Use a task scheduling framework (such as Quartz, Spring Scheduler) to schedule scheduled tasks. Scheduled task configurations can be saved in the database, and the system reads the configuration and executes tasks regularly.

[0054] Through the above steps, the data source, database, data table, data field and scheduled task in the business reporting items are automatically processed and analyzed, which simplifies the data processing process and reduces the risk of errors caused by manual intervention. At the same time, the system can flexibly handle multiple data sources and support dynamic scheduling of scheduled tasks, thereby improving the accuracy and consistency of reported data and significantly reducing the risk of data misreporting and omission.

[0055] S20, generating an SQL statement according to the data source, database, data table and data field specified in the business report item;

[0056] In this embodiment, the data source specified in the business report item is the storage location of the data, which may be a relational database, a NoSQL database, or other types of database systems. When generating SQL statements, first, it is necessary to confirm and connect to the specified data source based on the information in the report item to ensure that data can be obtained from the correct source.

[0057] The system can use database connection drivers such as JDBC and ODBC to connect to different data sources. If there are multiple data sources, you can use multi-data source configuration (such as Spring multi-data source support) to dynamically select and switch data sources.

[0058] The specified database refers to the specific database instance where the business reporting data is located. Different business reporting items may correspond to different databases. The system needs to confirm the specific database to be operated based on the information of the reporting item.

[0059] The system uses a database connection pool (such as HikariCP) to dynamically process connections with the database by parsing the database configuration parameters in the business report items. In the connection pool, connection processing is performed based on configuration information such as database URL, user name, and password.

[0060] Specific data tables need to be specified in the business report items. The system extracts relevant data by querying these data tables. The name and structure of the data table will affect the generation of SQL statements. The system needs to generate query statements based on the field structure in the data table.

[0061] The system parses the data table information in the submitted items, extracts the table name, and then inserts the table name into the SQL template when generating the SQL statement. The database metadata interface (such as DatabaseMetaData) can be used to verify whether the specified data table exists to ensure the correctness of the SQL statement.

[0062] Data fields are specific columns to be queried from the data table. When generating SQL statements, the system needs to select the fields to be queried based on the field information in the submitted items.

[0063] By parsing the field list in the submitted item, the SELECT clause in the SQL statement is generated, and the fields to be queried are selected. The system can dynamically verify the existence of the field to ensure that the specified field is consistent with the data table structure and avoid generating invalid SQL statements.

[0064] The process of generating SQL statements involves combining data sources, databases, data tables, and data fields into a complete query statement. The query statement is used to extract specified data from the database to meet the needs of business reporting.

[0065] The system builds an SQL query template (such as SELECT field FROM table WHERE condition) based on the information in the report item, fills the parsed data table and field name into the template, and forms the final SQL statement. For complex queries, the system can generate dynamic SQL statements based on the parameters in the report item, including conditional query, grouping, sorting and other functions.

[0066] Example: The headquarters of an insurance company needs to submit insurance business data to the regulatory agency on a regular basis. To this end, the business report item specifies the relevant information for query, including data source, database, data table, and data field. The report item requires extracting information about insurance policies and customers from the Insurance_DB database of the headquarters. The specific data table is policy_data, and the fields policy_id (policy number), customer_name (customer name), and issue_date (policy issuance date) are required to be extracted.

[0067] Through the above steps, the system can dynamically generate efficient SQL query statements based on the parameters specified in the business report items, and extract the required data from the specified data sources, databases, and data tables. This improves the system's data processing automation capabilities, avoids the risk of errors in manually writing SQL queries, and ensures the flexibility and accuracy of data extraction, supporting flexible processing of different data sources and cross-system data acquisition.

[0068] S30, regularly executing the SQL statement through the scheduled task to extract the business submission data of the corresponding submission batch from the specified data source and database;

[0069] In this embodiment, the scheduled task is used to control the execution time and frequency of SQL statements. The system triggers the execution of SQL statements at a specified time interval or at a specific time according to the scheduled task configuration in the business report item, ensuring that the business report data is extracted from the data source on time.

[0070] Use a task scheduling framework (such as Quartz or Spring Scheduler) to configure scheduled tasks. Scheduled tasks can be set to execute on a daily, weekly, monthly, or custom time interval. The task scheduling framework will periodically execute SQL queries according to the set time plan.

[0071] The system automatically executes the pre-generated SQL statement at the set time point to extract the business reporting data from the specified database. After the SQL statement is executed, the query result will be returned as the basis for the business reporting data.

[0072] The system triggers the database connection pool through a scheduled task and executes SQL queries from the database. SQL statement execution can be performed through standard JDBC or ORM frameworks (such as Hibernate) to ensure the efficiency and reliability of queries. During the execution process, the query can be logged, including the execution status, execution time, and query results.

[0073] Based on the definition of business reporting items, the system extracts data in batches from the specified data sources and databases. Each batch of data is filtered according to the time range or other conditions of the reporting item and distinguished from historical reporting data to avoid data duplication or omission.

[0074] The data is filtered by the WHERE condition in the SQL statement (such as time range, ID range, etc.) to ensure that the data extraction of each submitted batch is accurate. Batch data extraction can be improved through paging query (LIMIT, OFFSET, etc.) to improve efficiency, especially when processing large-scale data.

[0075] By executing SQL queries regularly through scheduled tasks, the system can automatically extract reporting data from designated data sources and databases on time, ensuring the regularity and accuracy of reporting tasks and reducing the risk of errors in human operations. The system automatically extracts data in batches to ensure that the reported data is timely, complete, and accurate, effectively avoiding data omissions and duplicate submissions.

[0076] S40, based on the business report data, analyzing the integrity, validity, uniqueness, accuracy and consistency of the data, and generating quality analysis results of corresponding business report items;

[0077] In this embodiment, integrity analysis refers to checking whether the business report data contains all necessary fields and whether there is any missing data. The integrity analysis is to ensure that all required fields in each record of the reported data are filled and meet the expected format requirements.

[0078] Use SQL statements or programming languages ​​to check whether there are records with empty fields. For example: SELECT*FROM policy_data WHERE customer_name IS NULL OR policy_id IS NULL. For mandatory fields, the system can mark which fields are mandatory according to business rules and automatically check the integrity of these fields.

[0079] Validity analysis refers to checking whether data values ​​comply with predefined business rules or format requirements, such as whether the format of the field is correct, whether the format of the date field meets the expected format, whether the numeric field is within the allowed range, etc.

[0080] Use regular expressions or SQL queries to check the correctness of the field format. For example: SELECT*FROM policy_dataWHERE NOT(policy_id REGEXP'^[AZ]{2}[0-9]{8}$'). Perform range checks on numeric fields. For example, the amount field cannot be a negative number, and the date field cannot be a future date.

[0081] Uniqueness analysis is to check whether there is duplication of unique identifiers in the data. Uniqueness is particularly important for certain fields (such as insurance policy number, ID number, etc.) because these fields should be unique in the data set.

[0082] Use SQL statements to check for duplicate records. For example, to check the uniqueness of the policy_id field: SELECT policy_id, COUNT(*) FROM policy_data GROUP BY policy_id HAVING COUNT(*)>1. Ensure uniqueness when storing data through uniqueness constraints (such as unique indexes) in the database.

[0083] Accuracy analysis is to verify whether the data conforms to the business logic. For example, the policy issuance date cannot be earlier than the customer's birth date, and the customer's age must be in line with the actual situation, etc. Accuracy analysis ensures that the data conforms to the actual business scenario.

[0084] Write check conditions according to the business logic. For example, to ensure that the policy issuance date is after the customer's birth date: SELECT * FROM policy_data WHERE issue_date < birth_date. For the check of numerical fields, the numerical range can be verified according to the business requirements.

[0085] Consistency analysis refers to checking whether the values of the same field are consistent in different data tables and different data sources. For example, whether the customer names with the same policy number are consistent in different data sources.

[0086] Verify the consistency of the same data field in multiple systems through cross-data table or cross-database queries. For example: SELECT a.policy_id, a.customer_name, b.customer_name

[0087] FROM policy_data a

[0088] JOIN external_policy_data b ON a.policy_id = b.policy_id

[0089] WHERE a.customer_name != b.customer_name;

[0090] Based on the above analysis dimensions, the system will generate the analysis results for each dimension, such as the pass rate for each dimension, the data entries that failed, etc., to form a complete quality analysis report for subsequent business decisions. The system will count the analysis results for each dimension, such as the integrity pass rate, the validity pass rate, etc., to generate a data report. Output the analysis results in a structured format (such as JSON or CSV) for subsequent analysis and display.

[0091] Through the multi-dimensional quality analysis system, enterprises can comprehensively and deeply detect the integrity, validity, uniqueness, accuracy and consistency of business reporting data to ensure high standards and reliability of data quality. It reduces the risk of misreporting and underreporting, provides reliable quality assurance for corporate regulatory reporting, and improves the automation level of data processing and business compliance.

[0092] S50, setting a threshold for early warning according to the historical data analysis results of the business report items;

[0093] In this embodiment, the trend and regularity of data quality are identified by analyzing the historical data of business reporting items. The historical data of key dimensions such as completeness, validity, uniqueness, accuracy and consistency are analyzed to obtain the historical performance of each dimension, providing data support for setting warning thresholds.

[0094] The system extracts the historical quality analysis results of each reporting item from the historical records, and uses data analysis tools (such as SQL, Python, etc.) to count and analyze the historical data. The performance of different dimensions can be analyzed by statistical methods (such as mean, standard deviation, quantile), and the basic data for setting warning thresholds can be obtained based on these analysis results. For example, the mean or fluctuation range of the data quality dimension within a certain period of time can be calculated.

[0095] Based on the historical data analysis results, the system sets warning thresholds for each data quality dimension. These thresholds are used to identify whether there are risks in the current data quality. When the quality analysis results are lower than the warning threshold, the system will trigger a warning.

[0096] The system uses statistical methods to set reasonable thresholds based on historical data trends. For example, quantiles (such as 20% quantile, 5% quantile) or standard deviations are used to determine the warning thresholds for each dimension. For some key reporting items, the thresholds can be set manually by the user or dynamically adjusted based on historical data.

[0097] Example: In financial data reporting, the system sets an early warning threshold based on the quality performance of customer transaction data in the past two years. The system sets the early warning threshold to 95% by analyzing the fluctuation range of the accuracy dimension in historical transaction data. When the newly extracted transaction data passes the accuracy analysis below 95%, the system automatically issues an early warning notification to remind relevant departments to check data quality issues.

[0098] Through analysis based on historical data, the system can set reasonable warning thresholds for each business report item to ensure that data quality issues can be identified and handled in a timely manner. It not only provides a dynamic and flexible threshold setting mechanism, but also enhances the system's sensitivity to data quality fluctuations, improves the ability to warn of potential data risks, and effectively reduces the risk of misreporting and underreporting caused by data quality issues.

[0099] S60, when the quality analysis result of the business report item is lower than the set threshold, an early warning is triggered, and the early warning information is sent to the business report terminal.

[0100] In this embodiment, the system will continuously monitor the quality analysis results of the business report items and compare the current quality analysis results with the pre-set threshold. When the analysis result of a data dimension (such as completeness, validity, uniqueness, accuracy or consistency) is lower than the set threshold, the system considers that the report item has quality risk.

[0101] Compare the current data quality analysis results (such as pass rate) with the preset threshold through SQL query or data analysis tools. For example: SELECT * FROM quality_evaluation WHERE completeness_rate < 80; the system regularly runs the quality analysis module, automatically detects submissions below the threshold, and marks them as abnormal. The system can use a monitoring framework (such as Prometheus, etc.) to regularly check the quality analysis results and automatically find data below the threshold.

[0102] When the quality analysis result of a business report item is found to be lower than the set threshold, the system immediately triggers the early warning mechanism, indicating that there is a problem with the data. The early warning mechanism can be triggered by generating an alarm within the system or notifying external systems and personnel through messages.

[0103] The way to trigger an early warning can be to generate an alarm log or generate an alarm notification through the monitoring system. The system can integrate alarm services (such as email notifications, SMS reminders, message push, etc.). When the quality analysis is below the threshold, an early warning message is automatically generated and sent to the designated recipient. The early warning information can include data quality dimensions, detailed reasons for being below the threshold, and analysis results of related data.

[0104] Once the warning information is generated, the system needs to send the information to the business reporting end or relevant personnel so that timely measures can be taken. The warning information contains the detailed results of the data quality analysis, helping the business reporting end to quickly identify data problems and make corrections.

[0105] The system sends warning information to the business reporting end through email, SMS or internal enterprise messaging systems (such as DingTalk, WeChat for Enterprise). The information should include the name of the reporting item, analysis dimension, analysis results and specific reasons for being below the threshold. The system can configure a target list for receiving warning information, including the person in charge of the business department, developers or processing personnel, to ensure that problems can be fed back and processed in a timely manner.

[0106] By monitoring the quality analysis results of business reporting items in real time, when the data quality is lower than the preset threshold, the system can quickly trigger an early warning and feedback the problem to the relevant departments. This mechanism enables data quality issues to be discovered and handled at an early stage, reducing compliance risks and business impacts caused by data misreporting and omissions. At the same time, the system automatically processes early warning information, avoiding delays caused by manual intervention and improving the accuracy and timeliness of reported data.

[0107] The present invention relates to the fields of big data technology and financial technology, and discloses a quality analysis and early warning method for business reporting data. By receiving and processing the reporting items submitted by the business reporting terminal, the specified data source, database, data table, data field and scheduled task information are extracted. Based on this information, an SQL statement is generated, and the execution of the SQL statement is triggered according to the scheduled task to extract the corresponding batch of business data from the database. The extracted data is subjected to quality analysis, and the analysis dimensions include integrity, validity, uniqueness, accuracy and consistency, and a quality analysis result is generated. According to the historical data analysis results, a threshold for triggering an early warning is set. When the quality analysis result is lower than the preset threshold, an early warning is triggered and relevant personnel are notified. The early warning information includes data quality dimensions, evaluation results and early warning levels. The present invention improves the accuracy and integrity of the reported data through automated processing and monitoring, effectively reduces the risk of misreporting and underreporting, improves data processing efficiency, enhances the early identification and processing capabilities of data quality problems, and ensures the standardization and compliance of the reporting tasks of the financial insurance industry.

[0108] In one embodiment, the above S20 includes:

[0109] S201, parsing the data source information specified in the business report item, determining the connection parameters of the database, and connecting to the specified data source;

[0110] S202, generating a corresponding SQL query template according to the data table and data fields in the business report item;

[0111] S203, extracting query conditions and query parameters from the business report items, wherein the query parameters are specific values ​​required in the query conditions, including query time values, and filling the query conditions and query parameters into the SQL template to form a complete SQL statement.

[0112] In this embodiment, the system first extracts the data source information from the business report item, including the type of data source (such as MySQL, PostgreSQL, etc.), database address, port number, user name, password and other connection parameters. After parsing this information, the system will use these parameters to establish a connection with the specified data source to ensure that the data source can be accessed correctly.

[0113] The system parses the JSON or XML formatted data in the submitted items, extracts the data source connection information, and uses standard drivers such as JDBC and ODBC to connect to the data source.

[0114] For example, after the system parses the database address and port number, it generates a database connection string, such as: StringconnectionString = "jdbc:mysql: / / database_address:3306 / database_name";

[0115] Use a connection pool (such as HikariCP) to handle database connections to ensure efficient and stable connections.

[0116] After connecting to the specified data source, the system will generate an SQL query template based on the table name and field information in the business report item. The template is a preliminary query structure that contains a SELECT clause for subsequent query conditions.

[0117] The system builds an SQL query template by parsing the data table name and field name in the report item. For example, if the table name specified in the report item is policy_data and the fields are policy_id and customer_name, the generated SQL template is: SELECT policy_id, customer_name FROM policy_data;

[0118] Templates can also contain placeholders for WHERE conditions, which are then filled with parameters.

[0119] The system further extracts query conditions and specific parameters from the business reporting items. Query conditions are abstract rules, such as time range or status filtering, while query parameters are specific values ​​used to execute these rules. Query conditions can be time ranges, field value ranges, etc., used to filter specific data. Query parameters are actual values ​​used to fill in placeholders in query conditions. In particular, in the case of querying time ranges, the system will use specific start and end times as parameters. In business reporting items, if you need to query data for a specific time period, such as "query transaction records for the past month", the system will set the query condition to "time range", and the specific start and end dates (such as "2023-01-01" and "2023-01-31") will be used as query parameters. These parameters will be used to replace placeholders in the SQL template and eventually generate executable query statements. After extracting these conditions and parameters, the system dynamically fills them into the previously generated SQL query template to form an executable complete SQL statement.

[0120] The query conditions and query parameters are integrated and filled into the SQL template to generate an executable SQL query. The SQL template is a predefined query statement that contains placeholders for dynamically inserting specific query parameters.

[0121] This embodiment analyzes the data source, data table, data field and query conditions in the business report items, and the system can automatically generate efficient SQL query statements, reduce the risk of errors in manually writing SQL, and ensure the accuracy and efficiency of data extraction. The system can flexibly handle multiple conditional queries, adapt to dynamic data requirements in different business scenarios, and greatly improve the automation level and reliability of reported data processing.

[0122] In one embodiment, the above S40 includes:

[0123] S401, counting data entries that meet the integrity standard, and generating integrity analysis scores based on the data entries that meet the integrity standard, wherein the integrity standard requires that the mandatory fields of the data entry are not empty and the data entry meets the predetermined requirements of the field;

[0124] S402, counting data entries that meet the validity criteria, and generating validity analysis scores based on the data entries that meet the validity criteria, wherein the validity criteria require that field values ​​of the data entries meet format specifications or meet predefined business rules;

[0125] S403, counting data entries that meet the uniqueness standard, and generating a uniqueness analysis score based on the data entries that meet the uniqueness standard, wherein the uniqueness standard requires that the unique identification field of the data entry does not have to be repeated in all data entries;

[0126] S404, counting data entries that meet the accuracy standard, and generating an accuracy analysis score based on the data entries that meet the accuracy standard, wherein the accuracy standard requires that the field value of the data entry is consistent with a predetermined business logic and range;

[0127] S405, counting data entries that meet the consistency standard, and generating a consistency analysis score based on the data entries that meet the consistency standard, wherein the consistency standard requires that the values ​​of the same field in the data entry in different data sources remain consistent;

[0128] S406, generating a quality analysis result based on the completeness score, validity score, uniqueness score, accuracy score and consistency score.

[0129] In this embodiment, the integrity standard requires that all required fields must be filled in and cannot have empty values, and the fields must comply with a predetermined format or rule. The system generates an integrity analysis score by counting data entries that meet the integrity requirements to measure the integrity of the data. SQL queries can be used to count which required fields in the data entries have empty values ​​or do not meet the requirements.

[0130] The system will count the data entries that meet the integrity criteria and compare them with the total number of entries that need to meet the integrity criteria to generate a completeness analysis score. The integrity criteria require that the required fields are not empty and meet the predetermined format requirements.

[0131] Score calculation method: Completeness score = (number of items that meet the completeness standard / total number of items that need to meet the completeness standard) * 100. For example, among 100 data that need to meet the completeness standard, if 90 data meet the completeness requirements, the completeness score is 90%.

[0132] The validity standard requires that the value of the field must comply with the specified format or business rules. For example, the format of the ID card number must comply with a specific regular expression, and the numeric field cannot exceed the specified range. The system generates a validity analysis score by counting the data entries that meet the validity standard. You can use regular expressions or SQL statements to check whether the data format meets the requirements.

[0133] The validity analysis score is generated by counting the data items that meet the validity criteria and comparing them with the total number of items that need to meet the validity criteria. The validity criteria require that the field value conform to the business rules or format specifications.

[0134] Score calculation method: validity score = (number of items that meet the validity criteria / total number of items that need to meet the validity criteria) * 100. For example, if 85 of the 100 data that need to meet the validity criteria meet the validity requirements, the validity score is 85%.

[0135] The uniqueness standard requires that the unique identification field in the data (such as insurance policy number, ID number, etc.) cannot be repeated in all data entries. The system generates a uniqueness analysis score by counting the data entries with no repeated unique identifiers. You can use SQL statements to check the repetition of unique identification fields.

[0136] The uniqueness analysis score is calculated by counting the data entries that meet the uniqueness criteria and comparing them to the total number of entries that need to meet the uniqueness criteria. The uniqueness criteria requires that the unique identification field does not repeat in all data entries.

[0137] Score calculation method: Uniqueness score = (number of entries that meet the uniqueness criteria / total number of entries that need to meet the uniqueness criteria) * 100.

[0138] Accuracy criteria require that the value of a field is consistent with business logic or business rules. For example, the issue date of an insurance policy cannot be earlier than the customer's date of birth. The system generates an accuracy analysis score by counting the data entries that meet the accuracy criteria. Write check conditions based on business logic, such as ensuring that the issue date of an insurance policy is later than the customer's date of birth.

[0139] The system generates an accuracy score by counting the data entries that meet the accuracy criteria and comparing them to the total number of entries that need to meet the accuracy criteria. The accuracy criteria require that the data values ​​meet the business logic and the predetermined range.

[0140] Score calculation method: Accuracy score = (number of items that meet the accuracy standard / total number of items that need to meet the accuracy standard) * 100.

[0141] The consistency standard requires that the values ​​of the same field in different data sources remain consistent. For example, customer names cannot be different in different database tables. The system generates a consistency analysis score by comparing data items across tables or data sources, counting consistent data entries. You can use cross-table queries to compare data fields from different sources.

[0142] The system will compare the same fields in different data sources, count the data entries that meet the consistency standards, and compare them with the total number of entries that need to meet the consistency standards to generate a consistency analysis score.

[0143] Score calculation method: Consistency score = (number of items that meet the consistency criteria / total number of items that need to meet the consistency criteria) * 100.

[0144] The system summarizes the analysis scores of each dimension and generates the final quality analysis result. This result is a multi-dimensional comprehensive analysis score, which is used to judge the overall quality of the data and serve as the basis for subsequent decision-making. The analysis scores of each dimension are weighted or simply averaged to generate a comprehensive quality analysis score. The system outputs the analysis results in a report format, such as JSON, CSV, etc., for subsequent analysis and presentation.

[0145] The final quality analysis result can be generated based on the score of each dimension. The result is usually the average of all dimension scores, or a comprehensive score calculated according to specific weights.

[0146] Score calculation method: Final score = (completeness score + validity score + uniqueness score + accuracy score + consistency score) / 5.

[0147] This embodiment conducts multi-dimensional analysis on the integrity, validity, uniqueness, accuracy and consistency of data, so that the system can comprehensively and accurately identify quality problems in business reporting data and generate detailed analysis results. This process has a high degree of automation, reduces the workload of manual inspection, improves the efficiency and accuracy of data processing, and provides a reliable basis for subsequent regulatory reporting and business decision-making.

[0148] In one embodiment, the above S50 includes:

[0149] S501, analyzing historical data of business reporting items, identifying trends and fluctuations of each data quality dimension, and determining a normal range of each data quality dimension, wherein the data quality dimensions include completeness, validity, uniqueness, accuracy, and consistency;

[0150] S501, according to the normal range, set multiple threshold levels for each data quality dimension, the multiple threshold levels include warning thresholds, risk warning thresholds and severe alarm thresholds, which are used to trigger the corresponding level of alarm mechanism.

[0151] In this embodiment, the system first extracts the relevant records of each data quality dimension (completeness, validity, uniqueness, accuracy, consistency) from the historical data of the business report items, and analyzes its performance trend and fluctuation range within a certain period of time. Through analysis, the performance of each data dimension under normal circumstances can be identified, and the normal range can be determined, so as to set the threshold of early warning later.

[0152] The system extracts scores of various dimensions from historical submission data and identifies the changing trends and fluctuation ranges of these scores through time series analysis. For example, the system can draw a trend chart by counting the scores of data quality dimensions (such as completeness) of each submission batch in the past three months to identify the fluctuation range of this dimension under normal circumstances.

[0153] After determining the normal range of each dimension, the system will set multiple threshold levels for each dimension. Each dimension sets multiple warning level thresholds based on its normal range, so that different levels of alarms can be triggered when data quality is abnormal.

[0154] The system sets multiple thresholds for each data dimension based on the analysis results of historical data and business needs. For example, the normal range of the integrity dimension is 85%-95%, and the system can set three thresholds for it: warning threshold 85%, risk warning threshold 80%, and severe alarm threshold 75%. The thresholds can be calculated by statistical methods (such as mean, quantile, standard deviation, etc.), or they can be manually adjusted according to business needs.

[0155] Multiple threshold levels for each data quality dimension are used to define different alert levels. When the data quality score is below a certain threshold, the system triggers an alert mechanism of the corresponding level. The early warning threshold is used to indicate minor anomalies, the risk warning threshold is used to indicate potential risks, and the severe warning threshold indicates serious quality issues.

[0156] The system sets three thresholds for each dimension: when the score is lower than the warning threshold, a general warning is triggered; when it is lower than the risk warning threshold, a moderate warning is triggered; when it is lower than the severe warning threshold, the highest level of warning is triggered. The alarm mechanism for each threshold level can be sent to relevant personnel via email, SMS, or other internal notification systems of the enterprise to handle data quality issues in a timely manner.

[0157] Through the analysis of historical data and threshold setting, the system can effectively identify the changing trend of data quality and set a reasonable early warning mechanism for each data quality dimension. The system triggers the corresponding early warning level according to different threshold levels, so that data anomalies can be discovered and handled in a timely manner, reducing the risks brought by data quality problems. At the same time, the setting of multiple threshold levels improves the flexibility and responsiveness of the system.

[0158] In one embodiment, the above S30 includes:

[0159] S301, obtaining the execution frequency and time of the scheduled task, and periodically triggering the execution of the SQL statement based on the execution frequency and time;

[0160] S302, when the scheduled task is triggered, connect to the specified data source and database and execute SQL statements;

[0161] S303, extracting the business reporting data of the corresponding reporting batch from the specified data source, and storing the business reporting data in the storage system;

[0162] S304, recording a log of each SQL statement execution, wherein the log includes the execution status, the number of extracted data items, and error information;

[0163] S305, when the SQL statement execution fails, an error log is recorded and an alarm notification is issued, and an automatic retry mechanism is enabled.

[0164] In this embodiment, the system obtains the configuration parameters of the scheduled task from the business report item, including the execution frequency (such as daily, weekly or monthly) and the execution time. Based on these parameters, the system will regularly trigger the execution of SQL statements to ensure that data is extracted from the specified data source on time.

[0165] Scheduled tasks can be configured through a scheduling framework (such as Quartz or Spring Scheduler). The system will periodically check the task plan and automatically trigger the execution of SQL statements at the preset time. Scheduled tasks can be set to a certain time period every day, a certain day of the week, or flexibly configured according to business needs.

[0166] When a scheduled task is triggered, the system automatically establishes a connection with the specified data source (such as connecting to MySQL, PostgreSQL and other databases). After the connection is established, the system executes query operations based on pre-generated SQL statements to extract business reporting data from the database.

[0167] The system uses JDBC or other database connection tools to establish a connection with the data source based on the database connection parameters (including address, port, user name and password). The execution of SQL statements and database interaction are carried out through a standardized database connection interface.

[0168] After the SQL statement is executed, the system extracts data that meets the query conditions from the database and stores the data in batches in a predetermined storage system, which may be local storage, cloud storage, or other data storage platforms.

[0169] After data extraction, the system will store the data in a structured format (such as CSV, JSON, XML, etc.) in a specified location according to the reporting batch to ensure that subsequent reporting and data analysis can proceed smoothly. The storage system can be an internal database, an external file system, or a cloud storage service.

[0170] The system needs to record the status of each SQL statement execution, including success or failure marks, the number of data extracted, and any error information. This log information is helpful for subsequent auditing and troubleshooting.

[0171] The system will generate a log file or record it in the log processing system, and the status of each execution (such as success, failure), the amount of data extracted, the execution time, etc. will be recorded. If an error occurs during SQL execution, the error log will record the cause and context of the error in detail.

[0172] When an SQL statement fails to execute, the system will generate an error log and immediately send out an alarm through the preset notification channel (such as email, SMS, and messaging platform) to remind relevant personnel to handle it. At the same time, the system will enable an automatic retry mechanism to re-execute the failed SQL query to ensure that the data can be successfully extracted.

[0173] The error log records the details of the execution failure, and the system sends the alert to relevant personnel through integrated alert services (such as email notification, SMS, and internal corporate messaging systems). The automatic retry mechanism attempts to re-execute the SQL statement multiple times through the set number of retries and intervals until it succeeds or reaches the set retry limit.

[0174] This embodiment uses a scheduled task to regularly execute SQL queries, so that the system can ensure that business reporting data is extracted and stored on time, reduce manual intervention, and improve the efficiency and reliability of data extraction. At the same time, the system automatically records the execution status of each task, provides detailed log information, and ensures the smooth completion of the task through an automatic retry mechanism in case of failure, avoiding the risk of data extraction failure.

[0175] In one embodiment, the above S60 includes:

[0176] S601, comparing the quality analysis results of the business report items with the set threshold value, and identifying the business report items below the threshold value;

[0177] S602, when it is identified that the quality analysis result of the business report item is lower than the set threshold, a warning of the corresponding level is triggered;

[0178] S603, generating warning information, wherein the warning information includes the data quality dimension that triggers the warning, the current quality analysis result, the corresponding threshold, the warning type and the warning time;

[0179] S604, sending the warning information to the business reporting end, and recording the warning information triggered each time.

[0180] In this embodiment, the system automatically compares the quality analysis results of the current business submission with the previously set threshold and identifies those submissions that are below the set threshold. Each quality dimension (such as completeness, validity, uniqueness, accuracy, consistency) has a corresponding threshold. When the score of a dimension is below the threshold, the system will mark the submission as having quality problems.

[0181] The system will calculate the quality analysis results regularly or in real time and compare them with the preset thresholds. If the evaluation result of a data quality dimension is lower than the preset threshold, the system will automatically identify the business submission item as "below the standard" and prepare for further processing.

[0182] Once the system identifies that the quality analysis result of a business report item is lower than the set threshold, the system will trigger different levels of warnings according to the severity of the threshold. Different warning levels (such as general warning, risk warning, and serious warning) correspond to different response strategies.

[0183] The system can automatically trigger different warning levels based on the gap between the evaluation results and the threshold. For example, when the quality analysis result is less than 10% of the threshold, a normal warning is triggered, and when it is less than 20%, a severe alarm is triggered. The setting of the warning level can be flexibly configured, and the system determines the warning classification standard based on business needs.

[0184] When the system triggers an early warning, it needs to generate complete early warning information. The early warning information should describe in detail the specific reasons for the early warning, including which quality dimension has a problem, the current quality evaluation score, the corresponding threshold, the type of early warning (such as ordinary warning, severe warning) and the time of the early warning.

[0185] The system will automatically generate warning information and record all relevant information, including: which dimension of quality issues triggered the warning, what is the current score, what is the threshold, the severity of the warning, and the specific time when the warning occurred. The generated warning information is saved in a structured format for subsequent analysis and processing.

[0186] Once the warning information is generated, the system will send it to the business reporting end so that relevant personnel or systems can receive and process it in time. In addition, the system will record each triggered warning information for future review and analysis.

[0187] The system sends warning information via email, SMS or enterprise messaging platforms (such as DingTalk, WeChat for Enterprise, etc.) to ensure that the business reporting end can obtain it in time. The system also records the detailed information of each warning in the internal log, including the warning type, triggering reason, time and sending status.

[0188] This embodiment monitors the quality analysis results and thresholds of business reporting items in real time. The system can trigger an early warning in time when data quality deteriorates, and automatically notify relevant departments to handle the problem. The detailed record and classification mechanism of each early warning help enterprises respond to data problems quickly, reducing the operational and compliance risks brought by data quality. The system's automated early warning function improves the reliability and response speed of regulatory reporting.

[0189] In one embodiment, the above S10 includes:

[0190] S101, receiving a business report item submitted by a business report terminal, parsing the business report item, and extracting information on data sources, databases, data tables, data fields, and scheduled tasks;

[0191] S102, processing the extracted data source, database, data table, data field and scheduled task information, including processing of data source connection parameters and maintenance and recording of database structure;

[0192] S103, verifying the validity of the data source connection and storing the extracted information in the processing system;

[0193] S104, updating the stored information according to the modified or newly added business reporting items submitted by the business reporting terminal.

[0194] In this embodiment, the system first receives a report item from the business reporting end. The report item contains detailed information related to the data source, including the type of data source (such as MySQL, Oracle, etc.), database name, table name, specific fields, and configuration of scheduled tasks. The system parses this information and extracts relevant parameters for subsequent operations and processing.

[0195] The system receives the report items submitted by the business reporting end through the API interface or files (such as JSON, XML format). The system parses the relevant configuration information of the data source, database, data table, data field, and scheduled task, and stores this information in the memory or database.

[0196] The system processes the extracted data source, database, table structure, fields, scheduled tasks, etc. In particular, the connection parameters of the data source (such as address, port, user name, password, etc.) will be processed securely, and the database structure and table field information also need to be recorded and maintained for subsequent queries, data extraction, and task execution.

[0197] The system stores the extracted data source connection information (such as IP address, port number, user name, password, etc.) in a secure database to ensure that these connection parameters can be accessed safely. The structure of the database table (field name, data type, etc.) will be recorded by the system for subsequent queries and to process changes or modifications to the table. The configuration of the scheduled task (execution time, frequency, etc.) will also be recorded to ensure the accurate execution of the scheduled task.

[0198] After extracting the connection information of the data source, the system must first verify the validity of these connection parameters to ensure that it can successfully connect to the specified data source. After the verification is passed, the system will store these valid configuration information in the processing system as the basis for subsequent data extraction.

[0199] The system verifies whether the data source connection is valid by testing the connection. The connection test can be completed by connecting to the database and executing a simple query (such as SELECT 1). After the verification is passed, the system stores the data source connection information, database structure and detailed configuration of the scheduled task in the central processing system to ensure that subsequent operations can be performed based on this information.

[0200] When the business reporting end modifies existing reporting items or adds new reporting items, the system will dynamically update the configuration information stored in the processing system based on these changes. The system not only supports the update of reporting items, but also supports the dynamic adjustment of the execution plan of scheduled tasks or the modification of database connection parameters.

[0201] The system compares the new submissions with the existing submissions and automatically identifies the modifications, such as changes in the database table structure, adjustments to scheduled tasks, or updates to data source connection information. The updated information dynamically replaces the stored configuration information, ensuring that the system's processing system always maintains the latest configuration information.

[0202] By analyzing and processing the business report items, the system can automatically extract and process important configuration information such as data sources, databases, tables, fields, and scheduled tasks to ensure the accuracy and timeliness of the reported data. At the same time, the system dynamically updates and processes the change information of the reported items, improving the flexibility and reliability of business reporting.

[0203] In one embodiment, a quality analysis and early warning device for business reporting data is provided, and the quality analysis and early warning device for business reporting data corresponds one-to-one with the quality analysis and early warning method for business reporting data in the above embodiment. Figure 3 , Figure 3 This is a functional module diagram of a preferred embodiment of the quality analysis and early warning device for business reporting data of the present invention. Business reporting processing module 10, SQL generation module 20, data extraction module 30, data quality analysis module 40, early warning threshold setting module 50 and early warning notification module 60. The detailed description of each functional module is as follows:

[0204] The business report processing module 10 is used to obtain and process the business report items submitted by the business report terminal, wherein the business report items specify the data source, database, data table, data field and scheduled task;

[0205] SQL generation module 20, used to generate SQL statements according to the data source, database, data table and data field specified in the business report item;

[0206] The data extraction module 30 is used to periodically execute the SQL statement through the scheduled task to extract the business submission data of the corresponding submission batch from the specified data source and database;

[0207] The data quality analysis module 40 is used to analyze the integrity, validity, uniqueness, accuracy and consistency of the data based on the business report data, and generate quality analysis results of the corresponding business report items;

[0208] The warning threshold setting module 50 is used to set the threshold for warning according to the historical data analysis results of the business report items;

[0209] The warning notification module 60 is used to trigger a warning when the quality analysis result of the business report item is lower than a set threshold, and send the warning information to the business report terminal.

[0210] In one embodiment, the SQL generation module 20 is specifically used to:

[0211] Parse the data source information specified in the business report item, determine the database connection parameters, and connect to the specified data source;

[0212] Generate corresponding SQL query templates based on the data tables and data fields in the business reporting items;

[0213] Extract query conditions and query parameters from the business report items, where the query parameters are specific values ​​required in the query conditions, including query time values, and fill the query conditions and query parameters into the SQL template to form a complete SQL statement.

[0214] In one embodiment, the data quality analysis module 40 is specifically used to:

[0215] Counting data entries that meet the integrity standard, and generating integrity analysis scores based on the data entries that meet the integrity standard, wherein the integrity standard requires that mandatory fields of the data entry are not empty and the data entry meets predetermined requirements of the field;

[0216] Counting data entries that meet the validity criteria, and generating validity analysis scores based on the data entries that meet the validity criteria, wherein the validity criteria require that field values ​​of the data entries meet format specifications or meet predefined business rules;

[0217] Counting data entries that meet a uniqueness criterion, and generating a uniqueness analysis score based on the data entries that meet the uniqueness criterion, wherein the uniqueness criterion requires that a unique identification field of a data entry does not have to be repeated in all data entries;

[0218] Counting data entries that meet the accuracy standard, and generating an accuracy analysis score based on the data entries that meet the accuracy standard, wherein the accuracy standard requires that the field value of the data entry is consistent with a predetermined business logic and range;

[0219] Counting data entries that meet the consistency standard, and generating a consistency analysis score based on the data entries that meet the consistency standard, wherein the consistency standard requires that the values ​​of the same field in the data entry in different data sources remain consistent;

[0220] A quality analysis result is generated based on the completeness score, validity score, uniqueness score, accuracy score, and consistency score.

[0221] In one embodiment, the warning threshold setting module 50 is specifically used to:

[0222] Analyze historical data of business submissions, identify trends and fluctuations in each data quality dimension, and determine the normal range for each data quality dimension, including completeness, validity, uniqueness, accuracy, and consistency;

[0223] A plurality of threshold levels are set for each data quality dimension according to the normal range, and the plurality of threshold levels include a warning threshold, a risk warning threshold and a severe alarm threshold, which are used to trigger an alarm mechanism of a corresponding level.

[0224] In one embodiment, the data extraction module 30 is specifically used to:

[0225] Obtaining the execution frequency and time of the scheduled task, and regularly triggering the execution of SQL statements based on the execution frequency and time;

[0226] When the scheduled task is triggered, connect to the specified data source and database and execute SQL statements;

[0227] Extracting the business reporting data of the corresponding reporting batch from the designated data source, and storing the business reporting data in the storage system;

[0228] Record the log of each SQL statement execution, which includes the execution status, number of data extracted and error information;

[0229] When a SQL statement fails to execute, an error log is recorded and an alert notification is issued, and an automatic retry mechanism is enabled.

[0230] In one embodiment, the warning notification module 60 is specifically used to:

[0231] Compare the quality analysis results of the business report items with the set thresholds to identify the business report items below the thresholds;

[0232] When the quality analysis result of the business report item is identified to be lower than the set threshold, an early warning of the corresponding level is triggered;

[0233] Generate warning information, the warning information including the data quality dimension that triggers the warning, the current quality analysis result, the corresponding threshold, the warning type and the warning time;

[0234] The warning information is sent to the business reporting terminal, and the warning information triggered each time is recorded.

[0235] In one embodiment, the business reporting processing module 10 is specifically used to:

[0236] Receive business reporting items submitted by the business reporting terminal, parse the business reporting items, and extract information about data sources, databases, data tables, data fields, and scheduled tasks;

[0237] Process the extracted data sources, databases, data tables, data fields and scheduled task information, including processing of data source connection parameters and maintenance and recording of database structures;

[0238] Verify the validity of the data source connection and store the extracted information into the processing system;

[0239] Update the stored information according to the modified or newly added business reporting items submitted by the business reporting end.

[0240] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external user terminal through a network connection. When the computer program is executed by the processor, it realizes the functions or steps of a service-side method for quality analysis and early warning of business reporting data.

[0241] In one embodiment, a computer device is provided. The computer device may be a user terminal, and its internal structure diagram may be as follows: Figure 5As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server through a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the user side of a method for quality analysis and early warning of business reporting data

[0242] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0243] Obtain and process the business reporting items submitted by the business reporting terminal, wherein the business reporting items specify the data source, database, data table, data field and scheduled task;

[0244] Generate SQL statements according to the data source, database, data table and data field specified in the business report item;

[0245] By means of the scheduled task, the SQL statement is executed regularly to extract the business submission data of the corresponding submission batch from the specified data source and database;

[0246] Based on the business reporting data, analyze the integrity, validity, uniqueness, accuracy and consistency of the data, and generate quality analysis results for the corresponding business reporting items;

[0247] Set thresholds for early warning based on historical data analysis results of business reporting items;

[0248] When the quality analysis result of the business report item is lower than the set threshold, an early warning is triggered, and the early warning information is sent to the business report terminal.

[0249] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0250] Obtain and process the business reporting items submitted by the business reporting terminal, wherein the business reporting items specify the data source, database, data table, data field and scheduled task;

[0251] Generate SQL statements according to the data source, database, data table and data field specified in the business report item;

[0252] By means of the scheduled task, the SQL statement is executed regularly to extract the business submission data of the corresponding submission batch from the specified data source and database;

[0253] Based on the business reporting data, analyze the integrity, validity, uniqueness, accuracy and consistency of the data, and generate quality analysis results for the corresponding business reporting items;

[0254] Set thresholds for early warning based on historical data analysis results of business reporting items;

[0255] When the quality analysis result of the business report item is lower than the set threshold, an early warning is triggered, and the early warning information is sent to the business report terminal.

[0256] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the user side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0257] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0258] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0259] It should be noted that if software tools or components other than those of the Company appear in the embodiments of the present application, they are only used for illustration and do not represent actual use. The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above-mentioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the above-mentioned embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A quality analysis and early warning method for business reporting data, characterized in that: The following steps are involved: Obtain and process the business report items submitted by the business report terminal, wherein the business report items specify the data source, database, data table, data field and scheduled task; Generate SQL statements according to the data source, database, data table and data field specified in the business report item; By means of the scheduled task, the SQL statement is executed regularly to extract the business submission data of the corresponding submission batch from the specified data source and database; Based on the business reporting data, analyze the integrity, validity, uniqueness, accuracy and consistency of the data, and generate quality analysis results for the corresponding business reporting items; Set thresholds for early warning based on historical data analysis results of business reporting items; When the quality analysis result of the business report item is lower than the set threshold, an early warning is triggered, and the early warning information is sent to the business report terminal.

2. The quality analysis and early warning method of business reporting data according to claim 1, characterized in that: Generate SQL statements based on the data source, database, data table and data field specified in the business report, including: Parse the data source information specified in the business report item, determine the database connection parameters, and connect to the specified data source; Generate corresponding SQL query templates based on the data tables and data fields in the business reporting items; Extract query conditions and query parameters from the business report items, where the query parameters are specific values ​​required in the query conditions, including query time values, and fill the query conditions and query parameters into the SQL template to form a complete SQL statement.

3. The quality analysis and early warning method of business reporting data according to claim 1, characterized in that: Based on the business reporting data, the integrity, validity, uniqueness, accuracy and consistency of the data are analyzed, and quality analysis results are generated, including: Counting data entries that meet the integrity standard, and generating integrity analysis scores based on the data entries that meet the integrity standard, wherein the integrity standard requires that mandatory fields of the data entry are not empty and the data entry meets predetermined requirements of the field; Counting data entries that meet the validity criteria, and generating validity analysis scores based on the data entries that meet the validity criteria, wherein the validity criteria require that field values ​​of the data entries meet format specifications or meet predefined business rules; Counting data entries that meet a uniqueness criterion, and generating a uniqueness analysis score based on the data entries that meet the uniqueness criterion, wherein the uniqueness criterion requires that a unique identification field of a data entry does not have to be repeated in all data entries; Counting data entries that meet the accuracy standard, and generating an accuracy analysis score based on the data entries that meet the accuracy standard, wherein the accuracy standard requires that the field value of the data entry is consistent with a predetermined business logic and range; Counting data entries that meet the consistency standard, and generating a consistency analysis score based on the data entries that meet the consistency standard, wherein the consistency standard requires that the values ​​of the same field in the data entry in different data sources remain consistent; A quality analysis result is generated based on the completeness score, validity score, uniqueness score, accuracy score, and consistency score.

4. The quality analysis and early warning method of business reporting data according to claim 1, characterized in that: According to the historical data analysis results of business reporting items, set the thresholds for early warning, including: Analyze historical data of business submissions, identify trends and fluctuations in each data quality dimension, and determine the normal range for each data quality dimension, including completeness, validity, uniqueness, accuracy, and consistency; A plurality of threshold levels are set for each data quality dimension according to the normal range, and the plurality of threshold levels include a warning threshold, a risk warning threshold and a severe alarm threshold, which are used to trigger an alarm mechanism of a corresponding level.

5. The quality analysis and early warning method of business reporting data according to claim 1, characterized in that: Through the scheduled task, the SQL statement is executed regularly to extract the business submission data of the corresponding submission batch from the specified data source and database, including: Obtaining the execution frequency and time of the scheduled task, and regularly triggering the execution of SQL statements based on the execution frequency and time; When the scheduled task is triggered, connect to the specified data source and database and execute SQL statements; Extracting the business reporting data of the corresponding reporting batch from the designated data source, and storing the business reporting data in the storage system; Record the log of each SQL statement execution, which includes the execution status, number of data extracted and error information; When a SQL statement fails to execute, an error log is recorded and an alert notification is issued, and an automatic retry mechanism is enabled.

6. The quality analysis and early warning method of business reporting data as claimed in claim 1, characterized in that: When the quality analysis result of the business report item is lower than the set threshold, an early warning is triggered, and the early warning information is sent to the business report terminal, including: Compare the quality analysis results of the business report items with the set thresholds to identify the business report items below the thresholds; When the quality analysis result of the business report item is identified to be lower than the set threshold, an early warning of the corresponding level is triggered; Generate warning information, the warning information including the data quality dimension that triggers the warning, the current quality analysis result, the corresponding threshold, the warning type and the warning time; The warning information is sent to the business reporting terminal, and the warning information triggered each time is recorded.

7. The quality analysis and early warning method of business reporting data according to claim 1, characterized in that: Obtain and process the business report items submitted by the business report terminal, in which the data source, database, data table, data field and scheduled task are specified, including: Receive business reporting items submitted by the business reporting terminal, parse the business reporting items, and extract information about data sources, databases, data tables, data fields, and scheduled tasks; Process the extracted data sources, databases, data tables, data fields and scheduled task information, including processing of data source connection parameters and maintenance and recording of database structures; Verify the validity of the data source connection and store the extracted information into the processing system; Update the stored information according to the modified or newly added business reporting items submitted by the business reporting end.

8. A quality analysis and early warning device for business reporting data, characterized in that: The quality analysis and early warning device of the business reporting data includes: The business report processing module is used to obtain and process the business report items submitted by the business report terminal, wherein the business report items specify the data source, database, data table, data field and scheduled task; An SQL generation module is used to generate SQL statements according to the data source, database, data table and data field specified in the business report item; A data extraction module is used to periodically execute the SQL statement through the scheduled task to extract the business submission data of the corresponding submission batch from the specified data source and database; A data quality analysis module, used to analyze the integrity, validity, uniqueness, accuracy and consistency of the data based on the business report data, and generate quality analysis results for the corresponding business report items; The warning threshold setting module is used to set the threshold for warning according to the historical data analysis results of the business report items; The early warning notification module is used to trigger an early warning when the quality analysis result of the business report item is lower than a set threshold, and send the early warning information to the business report terminal.

9. A computer device, characterized in that: The computer device includes a memory, a processor, and a quality analysis and early warning program for business reporting data stored in the memory and executable on the processor. When the quality analysis and early warning program for business reporting data is executed by the processor, the steps of the quality analysis and early warning method for business reporting data as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The storage medium stores a quality analysis and early warning program for business reporting data, which, when executed by the processor, implements the steps of the quality analysis and early warning method for business reporting data as described in any one of claims 1-7.

Images