Report data anomaly monitoring and quality assessment system, method and electronic equipment

Abstract

The present invention provides a report data anomaly monitoring and quality assessment system, method, and electronic device, relating to the field of data processing technology. The system includes: a data acquisition module for acquiring an enterprise's report data and analyzing it to obtain data points corresponding to each data type in the report data; an anomaly point analysis module for analyzing the data points and, in combination with preset judgment rules corresponding to the data type, determining whether any of the data points contain anomalies; an impact path tracing module for tracing the path of anomalies using a path analysis algorithm when an anomaly exists, determining the corresponding propagation path; an affected data analysis module for analyzing the degree of impact of data points in the propagation path to determine the affected data points; and a data assessment module for assessing the overall quality level of the report data based on the affected data points combined with the anomaly points, and generating a quality assessment report. The present invention improves the accuracy of the quality assessment results of the report data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a report data anomaly monitoring and quality assessment system, method and electronic equipment. Background Art

[0002] In modern enterprise operations, monitoring and assessing report data anomalies is crucial for ensuring data reliability and supporting effective decision-making. This typically relies on statistical methods to analyze the local characteristics of each data point, such as calculating the mean and standard deviation, to identify anomalies in the data.

[0003] Related technologies, which identify anomalies based solely on local features, ignore the potential impact of a data point as an outlier while also ignoring its impact on other data. Even if the affected data remains within the set threshold, its actual business significance may have deviated, leading to omissions and false positives, which can severely impact the quality assessment of report data. Summary of the Invention

[0004] The problem solved by the present invention is how to improve the accuracy of the quality assessment results of report data.

[0005] To solve the above problems, the present invention provides a report data anomaly monitoring and quality assessment system, method and electronic equipment.

[0006] In a first aspect, the present invention provides a report data anomaly monitoring and quality assessment system, comprising:

[0007] A data acquisition module is used to acquire the enterprise's report data and analyze the report data to obtain data points corresponding to each data type in the report data;

[0008] An outlier analysis module, configured to analyze the data points of the data type and determine whether there are outliers in the data points in combination with a preset judgment rule corresponding to the data type;

[0009] An impact path tracking module is used to, when the abnormal point exists in the data points, track the path of the abnormal point by using a path analysis algorithm to determine the propagation path corresponding to the abnormal point;

[0010] An affected data analysis module, configured to analyze the degree of impact of the data points in the propagation path and determine the affected data points;

[0011] A data evaluation module is used to evaluate the overall quality level of the report data based on the affected data points in combination with the abnormal points, and generate a quality evaluation report for the report data based on the overall quality level.

[0012] Optionally, the data acquisition module is specifically used to:

[0013] Obtaining initial report data from the database of the enterprise;

[0014] Performing format conversion and preprocessing on the initial report data to obtain standardized report data;

[0015] Identify the data according to the metadata information of the report data to obtain multiple data types in the report data;

[0016] The data points corresponding to the data type are extracted according to the data type.

[0017] Optionally, the outlier analysis module is specifically configured to:

[0018] Determining the preset judgment rule according to the data type, wherein the preset judgment rule includes a statistical sub-rule, a business logic sub-rule, and a machine learning sub-rule;

[0019] screening the data points of the data type according to the statistical sub-rules, the business logic sub-rules, and the machine learning sub-rules in sequence;

[0020] Determine whether there are abnormal points in the data points of the data type according to the screening result.

[0021] Optionally, the outlier analysis module is further configured to:

[0022] Determining, based on the statistical sub-rules, whether there are any deviating data points in the data points that exceed a preset normal distribution;

[0023] If the deviated data point exists in the data points, determining whether the deviated data point exceeds the business scope corresponding to the data type according to the business logic sub-rule;

[0024] If the deviated data point exceeds the business scope, performing an in-depth analysis on the deviated data point according to the machine learning sub-rule to obtain an abnormality level of the deviated data point;

[0025] According to the abnormality level, it is determined whether the deviated data point is the abnormal point.

[0026] Optionally, the impact path tracing module is specifically configured to:

[0027] Obtaining a business association rule and a data association rule corresponding to the data type of the abnormal point;

[0028] Add the abnormal point as the initial node to the propagation path queue;

[0029] Determining a downstream associated node of the initial node according to the business association rule and the data association rule;

[0030] Adding the downstream associated nodes to the propagation path queue in sequence to obtain a preliminary propagation path;

[0031] The preliminary propagation path is recursively expanded until no further expansion is possible, thereby obtaining the propagation path.

[0032] Optionally, the impact path tracing module is further configured to:

[0033] Determining, based on the business association rules, the node corresponding to the subsequent link of the initial node in the business process;

[0034] Determining, according to the data association rule, nodes corresponding to relevant data points of the initial node in the data structure;

[0035] The node corresponding to the subsequent link and the node corresponding to the relevant data point are used as the downstream associated nodes of the initial node.

[0036] Optionally, the affected data analysis module is specifically configured to:

[0037] According to the propagation path, acquiring the data point in the propagation path;

[0038] Performing an impact degree analysis on the data points to obtain an impact degree value of each data point;

[0039] Determining the affected data points according to the affected degree value;

[0040] The affected data points are marked, and a set of affected data points is generated.

[0041] Optionally, the data evaluation module is specifically configured to:

[0042] Establish a data quality assessment model;

[0043] Determining the severity of the outlier and each of the affected data points and the impact range of the outlier using the data quality assessment model;

[0044] Performing an overall quality assessment on the report data using the severity value of the outlier and each of the affected data points, as well as the impact range of the outlier;

[0045] Determine the abnormality level of the report data based on the evaluation result, and generate a list of abnormality details of the report data;

[0046] The quality assessment report is generated based on the abnormality details list and the quality score.

[0047] In a second aspect, the present invention provides a method for monitoring and assessing report data anomalies, including:

[0048] Obtaining enterprise report data, and analyzing the report data to obtain data points corresponding to each data type in the report data;

[0049] Analyze the data points of the data type, and determine whether there are abnormal points in the data points in combination with a preset judgment rule corresponding to the data type;

[0050] When the abnormal point exists in the data points, the path of the abnormal point is tracked by a path analysis algorithm to determine the propagation path corresponding to the abnormal point;

[0051] Performing an impact analysis on the data points in the propagation path to determine the affected data points;

[0052] The overall quality level of the report data is evaluated based on the affected data points in combination with the abnormal points; and a quality evaluation report of the report data is generated based on the overall quality level.

[0053] In a third aspect, the present invention provides an electronic device comprising a memory and a processor;

[0054] The memory is used to store computer programs;

[0055] The processor is used to implement the above-mentioned report data anomaly monitoring and quality assessment method when executing the computer program.

[0056] The report data anomaly monitoring and quality assessment system, method, and electronic device of the present invention acquire enterprise report data through a data acquisition module and analyze the report data to obtain data points corresponding to each data type, laying the foundation for subsequent analysis. The outlier analysis module analyzes the data points of each data type and, in conjunction with pre-set judgment rules corresponding to the data type, determines whether any outliers exist within the data points, preliminarily identifying local anomalies. If an outlier exists, the impact path tracing module uses a path analysis algorithm to trace the outlier path and determine the corresponding propagation path. Tracing the propagation path reveals other data points that may be affected by the outlier, not just the outlier itself. The affected data analysis module analyzes the degree of impact of data points along the propagation path, identifies the affected data points, and further explores the actual impact of the anomaly on other data, avoiding the problem of focusing solely on local features while ignoring overall correlations. The data assessment module assesses the overall quality of the report data based on the affected data points combined with the outliers and generates a quality assessment report. This results in a more comprehensive and accurate assessment, comprehensively considering the mutual influence of data.

[0057] The present invention comprehensively considers the relationship between data, from the discovery of anomalies to the tracing of impact paths, and then to the analysis of affected data. Compared with traditional statistical methods based only on local features, it can effectively avoid omissions and false positives, thereby greatly improving the accuracy of report data quality assessment results. At the same time, it not only focuses on the anomalies of the data points themselves, but also on the impact of anomalies on other data, making the assessment of report data quality more comprehensive and more realistically reflecting the reliability and effectiveness of data in actual business scenarios. Accurate quality assessment results can provide more powerful support for corporate decision-making. It effectively solves the problems of omissions and false positives caused by judging anomalies only from local features in traditional technologies, and improves the accuracy of report data quality assessment results. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a structural block diagram of a report data anomaly monitoring and quality assessment system according to an embodiment of the present invention;

[0059] Figure 2 Flowchart of a method for monitoring report data anomalies and assessing quality according to an embodiment of the present invention;

[0060] Figure 3 2 is a structural block diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] To make the above-mentioned objects, features, and advantages of the present invention more clearly understood, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0062] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0063] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to"; the term "based on" means "based at least in part on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments"; the term "optionally" means "optional embodiments". The relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0064] It should be noted that the modifications of "one" and "multiple" mentioned in the present invention are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0065] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0066] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0067] Combine Figure 1 As shown, an embodiment of the present invention provides a report data anomaly monitoring and quality assessment system, including:

[0068] The data acquisition module is used to acquire the enterprise's report data and analyze the report data to obtain data points corresponding to each data type in the report data.

[0069] Specifically, the data acquisition module, as the foundation of the entire system, collects report data from multiple data sources within the enterprise, including enterprise resource planning (ERP) systems, financial databases, and business operations systems. The data acquisition module communicates with these data sources through data interfaces (such as APIs and database connections), acquiring report data periodically or in real time. After acquiring the data, the module performs preliminary cleaning and preprocessing, including removing duplicate data, correcting data format errors, and filling in missing values ​​to ensure data integrity and consistency. The preprocessed report data is then analyzed to identify the data points corresponding to each data type in the report. For example, in a financial statement, assets, liabilities, revenue, and costs are all different data types, each corresponding to a series of specific data points, such as the amount of fixed assets or the amount of current liabilities. This distinction and identification of data types and data points provides a clear data structure for subsequent outlier analysis.

[0070] The outlier analysis module is used to analyze the data points of the data type and determine whether there are outliers in the data points in combination with the preset judgment rules corresponding to the data type.

[0071] Specifically, the outlier analysis module applies a variety of data analysis techniques and algorithms to detect anomalies in data points of each data type. First, preset judgment rules are set for each data type based on its characteristics. These preset judgment rules can be based on statistical methods, such as using statistical indicators such as the mean, standard deviation, and percentiles as the basis for judgment. For example, for a particular cost data item, if its value falls outside the range of the mean calculated over a period of time plus or minus three standard deviations, the data point may be preliminarily identified as an outlier. Furthermore, specific thresholds or conditions can be set based on business rules, such as the company's historical experience and business logic. For example, for sales data, if a particular day's sales drop significantly (e.g., by more than 50%) compared to the previous day or the same period, and there is no reasonable business explanation (such as promotional activities, holidays, etc.), it may be considered an anomaly. During the analysis process, the outlier analysis module examines each data point one by one, judging it according to the preset judgment rules, ultimately determining whether there is an outlier. These outliers are then recorded to provide clues for subsequent impact path tracing.

[0072] The impact path tracking module is used to track the path of the abnormal point when the abnormal point exists in the data point through a path analysis algorithm to determine the propagation path corresponding to the abnormal point.

[0073] Specifically, after the Outlier Analysis Module detects an outlier, the Impact Path Tracing Module applies a path analysis algorithm to determine the outlier's propagation path. This path analysis algorithm constructs a data dependency graph based on the enterprise's data flow and business logic. For example, in a manufacturing enterprise, an anomaly in raw material procurement costs may propagate along a path such as "raw material procurement - production and processing - product costs - sales costs," impacting data at each subsequent stage. In practice, the path analysis algorithm starts with the identified outlier and, based on data relationships (such as data calculation relationships and the order of precedence in the business process), gradually traces the potentially affected data points. For example, if an anomaly in a financial statement involves depreciation calculations, the anomaly may propagate along the path of "asset original value - depreciation expense - accumulated depreciation - net assets." Using this algorithm, the Impact Path Tracing Module comprehensively and systematically determines the outlier's propagation path within the report data, revealing potential connections between the outlier and other data points, and providing direction and scope for subsequent analysis of the affected data.

[0074] The affected data analysis module is used to analyze the degree of influence of the data points in the propagation path and determine the affected data points.

[0075] Specifically, the affected data analysis module establishes an impact assessment model based on the relationships between different data points and the characteristics of data propagation. For example, data points that are directly dependent on outliers may be more affected; while data points that are indirectly dependent or related to outliers through multi-layer computing relationships may be relatively less affected due to factors such as the attenuation of data propagation. During the analysis process, a variety of methods can be used to quantify the degree of impact, such as calculating the deviation rate of data points (the deviation from the expected value under normal circumstances) and correlation analysis (analyzing the strength of the correlation between data points and outliers). Through these analyses, it is possible to accurately determine which data points are affected by outliers and the extent of the impact, thereby accurately identifying the affected data points from the large amount of data, providing a key basis for further evaluating the overall quality of the report data.

[0076] A data evaluation module is used to evaluate the overall quality level of the report data based on the affected data points in combination with the abnormal points, and generate a quality evaluation report for the report data based on the overall quality level.

[0077] Specifically, the data assessment module comprehensively considers outliers and affected data points, assessing the overall quality of report data from multiple dimensions. During the assessment process, factors such as the severity of the outliers and the range and extent of the affected data points are quantified and integrated based on pre-defined quality assessment indicators and a weighting system. For example, the severity of an outlier can be scored based on its deviation from the normal range; the range of affected data points can be quantified based on the number of data types and data points involved; and the extent of the impact can be measured based on indicators such as the deviation rate derived from the previous analysis. Through weighted calculations and other methods, an assessment of the overall quality of the report data is obtained. Based on this assessment result, the system generates a detailed and accurate quality assessment report. This report not only describes the overall quality level but also includes specific information about the outliers (such as their location, type, and severity), the distribution of affected data points (including the types and specific data points involved), and the basis and methods for the quality assessment. Furthermore, the report can provide recommendations for improvement, such as further investigation of the outliers and affected data points and data correction measures, providing strong support for enterprises to optimize data management and decision-making.

[0078] The report data anomaly monitoring and quality assessment system of the present invention uses a data acquisition module to acquire enterprise report data and analyze the report data to obtain data points corresponding to each data type, laying the foundation for subsequent analysis. The outlier analysis module analyzes the data points of each data type and, in conjunction with pre-set judgment rules corresponding to the data type, determines whether any outliers exist within the data points, initially identifying local anomalies. If an outlier exists, the impact path tracing module uses a path analysis algorithm to trace the path of the outlier and determine the corresponding propagation path. Tracing the propagation path reveals other data points that may be affected by the outlier, not just the outlier itself. The affected data analysis module analyzes the degree of impact of data points along the propagation path, identifies the affected data points, and further explores the actual impact of the anomaly on other data, avoiding the problem of focusing solely on local features while ignoring overall correlations. The data assessment module assesses the overall quality of the report data based on the affected data points combined with the outliers and generates a quality assessment report. This results in a more comprehensive and accurate assessment, comprehensively considering the mutual influence of data.

[0079] The present invention comprehensively considers the relationship between data, from the discovery of anomalies to the tracing of impact paths, and then to the analysis of affected data. Compared with traditional statistical methods based only on local features, it can effectively avoid omissions and false positives, thereby greatly improving the accuracy of report data quality assessment results. At the same time, it not only focuses on the anomalies of the data points themselves, but also on the impact of anomalies on other data, making the assessment of report data quality more comprehensive and more realistically reflecting the reliability and effectiveness of data in actual business scenarios. Accurate quality assessment results can provide more powerful support for corporate decision-making. It effectively solves the problems of omissions and false positives caused by judging anomalies only from local features in traditional technologies, and improves the accuracy of report data quality assessment results.

[0080] Optionally, the data acquisition module is specifically used to:

[0081] Obtaining initial report data from the database of the enterprise;

[0082] Performing format conversion and preprocessing on the initial report data to obtain standardized report data;

[0083] Identify the data according to the metadata information of the report data to obtain multiple data types in the report data;

[0084] The data points corresponding to the data type are extracted according to the data type.

[0085] Specifically, the data acquisition module first establishes a connection with the enterprise database to obtain initial report data. In embodiments of the present invention, this can be achieved through a variety of methods, such as using database connection protocols (such as JDBC, ODBC, etc.) or by invoking an internal enterprise API. Specifically, the system establishes communication with the database server using the corresponding database driver based on pre-configured database connection information (including database address, port number, username, password, etc.). Then, by executing SQL queries, it extracts report data from the specified database table or view. This report data is stored in the enterprise's financial database, business operations database, or other related databases, covering information from various business areas, such as financial statements, sales data, inventory data, production data, and so on.

[0086] Since the initial report data obtained may come from different data sources and have various data formats. For example, some data may be stored in the form of a text file, in which data fields are separated by commas; some data may be stored in XML or JSON format. In order to unify the data format and facilitate subsequent processing, the system needs to convert data in these different formats. For data stored in the form of text files, in this embodiment, the format rules of the text file are parsed, the data is read line by line, and the data is split into separate fields according to the field delimiters, and then these fields are stored in a unified data structure, such as a data table or data frame. For data in XML or JSON format, the corresponding parsing library is used to parse the hierarchical structure of the data, extract the data elements, and convert them into a flat format or structured format consistent with other data.

[0087] After format conversion, data is preprocessed to improve data quality. This preprocessing step includes data cleaning and data validation. Data cleaning primarily involves removing duplicate data, correcting data format errors (such as inconsistent date formats or incorrect numeric formats), and filling missing values. For example, duplicate records can be identified and removed by comparing unique identifiers or other key fields. Missing values ​​can be filled using appropriate methods based on business rules and data characteristics, such as using the mean, median, mode, or other reasonable estimates. Data validation verifies that data complies with business rules and data integrity constraints. For example, it verifies that data falls within a reasonable range (e.g., age cannot be negative, sales cannot exceed the company's production capacity), and that logical relationships between data are correct (e.g., inventory quantity cannot exceed purchase quantity). For data that does not comply with validation rules, the system records the relevant error information and handles it according to pre-set processing strategies, such as notifying data management personnel to make corrections or automatically correcting the data.

[0088] Metadata information for report data is descriptive information about the data, including the data name, data type definition, data source, and data meaning. The system uses metadata to identify different data types within report data. For example, metadata might define a field in a report as "sales," with a numeric data type and a unit of yuan; and another field as "sales date," with a date data type. During implementation, the system reads the report data's metadata file or retrieves metadata information from a database and parses the report data based on these metadata definitions. By analyzing metadata information such as the name and data type of each data field, the system can accurately identify multiple data types within report data, such as financial data types (including assets, liabilities, revenue, and costs), sales data types (including sales revenue, sales volume, and sales regions), and production data types (including output, raw material consumption, and production efficiency).

[0089] After determining the various data types in the report data, the system extracts the corresponding data points based on each data type's definition. For each data type, the system traverses the report data, searching for records that match the data type's definition. For example, for the "sales" data type, the system searches all sales-related fields in the sales report data and extracts the values ​​of these fields as the data points corresponding to that data type. Furthermore, the extracted data points can be stored in a separate data structure, such as a dictionary or list, to facilitate subsequent analysis and processing.

[0090] In this optional embodiment, by obtaining the initial report data from the enterprise database and performing format conversion and preprocessing, the integration and unification of data from different sources and in different formats is achieved, providing a consistent data basis for subsequent exception monitoring and quality assessment. This avoids data analysis errors or difficulties caused by inconsistent data formats, and improves the efficiency and accuracy of data processing. The data cleaning and verification steps in the preprocessing process effectively improve the quality of the report data; removing duplicate data and correcting format errors reduce data redundancy and inconsistency, and filling in missing values ​​and data verification ensure the integrity and accuracy of the data, so that subsequent exception analysis and quality assessment can be carried out on a more reliable data basis, improving the credibility of the assessment results.

[0091] Using metadata to identify data types accurately divides report data into multiple data types, providing clear data categories for subsequent targeted analysis. This enables the system to more accurately detect anomalies and analyze affected data points based on different data characteristics and business rules, improving the accuracy and effectiveness of anomaly monitoring. Extracting data points based on data types allows for rapid location and acquisition of the specific data required for analysis, improving system processing efficiency. This saves time and computing resources for subsequent operations such as anomaly analysis and impact path tracing, enabling more efficient operation of the entire report data anomaly monitoring and quality assessment system.

[0092] Optionally, the outlier analysis module is specifically configured to:

[0093] Determining the preset judgment rule according to the data type, wherein the preset judgment rule includes a statistical sub-rule, a business logic sub-rule, and a machine learning sub-rule;

[0094] screening the data points of the data type according to the statistical sub-rules, the business logic sub-rules, and the machine learning sub-rules in sequence;

[0095] Determine whether there are abnormal points in the data points of the data type according to the screening result.

[0096] Specifically, the outlier analysis module first determines the corresponding preset judgment rules based on different data types. The preset judgment rules include statistical sub-rules, business logic sub-rules, and machine learning sub-rules.

[0097] Specifically, for various data types, statistical sub-rules calculate statistical indicators such as mean, standard deviation, and percentiles based on historical data to define the normal range for the data. For example, for normally distributed data, the normal range can be defined as the mean ± 3 times the standard deviation; for asymmetrically distributed data such as sales data, percentiles can be used to define the normal range. Specific business logic sub-rules are developed based on the company's business characteristics and experience. For example, in financial data, costs should not exceed revenue; for inventory data, month-end inventory levels should be within the safety stock range. These rules, based on business realities, provide a business basis for identifying outliers. Machine learning sub-rules utilize machine learning algorithms (such as isolation forests and support vector machines) to train models to distinguish between normal and abnormal data patterns. Taking production data as an example, isolation forests can be trained based on historical production data to identify outliers that deviate from normal production patterns.

[0098] Statistical sub-rules are used to perform a preliminary screening of data points. If a data point falls outside the set statistically normal range, such as exceeding ±3 standard deviations from the mean, it is marked as a suspected outlier. The pre-screened data points are then fed into the business logic sub-rules. If the cost data marked as suspected anomaly is still higher than the revenue data, it is further confirmed as a business logic outlier. The screened data points are then fed into a machine learning model, such as the isolation forest model, which makes a judgment based on the data point's characteristics and historical patterns. If the probability of a data point deviating from the normal pattern exceeds a threshold, it is identified as a machine learning outlier.

[0099] The results of statistical, business logic, and machine learning screening are combined to determine whether any data points contain outliers. If a data point is determined to be anomaly based on statistical, business logic, or machine learning screening, it is considered an outlier. For example, if a data point exceeds the normal range in statistical screening and violates business rules in business logic screening, it is directly determined to be an outlier. Weights are also assigned to different rule types based on their strictness and importance. For example, a weight of 0.5 can be set for business logic sub-rules, and 0.25 can be set for statistical sub-rules and machine learning sub-rules respectively. After calculating the combined weights, if the total score exceeds the threshold, the point is determined to be an outlier.

[0100] In this optional embodiment, by combining the three sub-rules of statistics, business logic, and machine learning, the statistical characteristics, business characteristics, and data patterns of the data are comprehensively considered, thus avoiding misjudgments and missed judgments caused by a single rule and improving the accuracy of outlier judgments. The machine learning sub-rules enable the system to automatically learn and update based on historical data, adapt to changes in data distribution and business logic, and enhance the system's adaptability to new data and new business situations. Screening data points from different angles makes outlier judgments more comprehensive and in-depth, providing a reliable basis for subsequent impact path tracing and affected data analysis, and helping to more accurately assess the overall quality of report data.

[0101] Optionally, the outlier analysis module is further configured to:

[0102] Determining, based on the statistical sub-rules, whether there are any deviating data points in the data points that exceed a preset normal distribution;

[0103] If the deviated data point exists in the data points, determining whether the deviated data point exceeds the business scope corresponding to the data type according to the business logic sub-rule;

[0104] If the deviated data point exceeds the business scope, performing an in-depth analysis on the deviated data point according to the machine learning sub-rule to obtain an abnormality level of the deviated data point;

[0105] According to the abnormality level, it is determined whether the deviated data point is the abnormal point.

[0106] Specifically, the outlier analysis module uses statistical principles to identify outliers. First, historical data for each data type in the report data is analyzed to calculate key statistical indicators. For data types that conform to a normal distribution, the mean (μ) and standard deviation (σ) are calculated. Based on empirical rules (e.g., 99.7% of data are concentrated within the mean ±3σ range), data points outside this range are preliminarily identified as outliers. For example, for a company's monthly product sales data, the mean is calculated to be 1 million yuan and the standard deviation is 100,000 yuan. If monthly sales reach 1.3 million yuan (100 + 3 × 10) or above, or fall below 700,000 yuan (100 - 3 × 10), the data point is considered an outlier. For data that does not conform to a normal distribution, such as skewed customer complaint data, the system uses percentile analysis. The 25th percentile (Q1) and the 75th percentile (Q3) are usually set, and the interquartile range (IQR=Q3-Q1) is calculated. Data points lower than Q1-k×IQR (k is generally 1.5 or 3) or higher than Q3+k×IQR are judged as deviating data points.

[0107] If a deviating data point exists, the outlier analysis module further determines, based on business logic sub-rules, whether it exceeds the business scope corresponding to the data type. The business scope is a reasonable range set based on the company's actual business operation rules and experience. Taking cost data as an example, the company's business logic stipulates that the cost of a product should not exceed 70% of its sales price. Suppose a product's sales price is 100 yuan, and its cost data has a deviating data point of 80 yuan. By calculating 80 ÷ 100 = 80% > 70%, this deviating data point exceeds the business scope. Another example is inventory data. The company stipulates that the end-of-month inventory of a product should not be less than the safety stock of 100 pieces. If a deviating data point of 50 pieces occurs, it also exceeds the lower limit of the business scope.

[0108] For data points that deviate from the business scope, the module uses a model trained based on machine learning sub-rules for in-depth analysis to determine their anomaly level. First, the system collects a large amount of historical data, including both normal and annotated anomaly data. Feature engineering is then performed on this data to extract data features that help distinguish anomalies. For production data, for example, these features might include production time, raw material usage, and equipment operating parameters. Next, an appropriate machine learning algorithm, such as a support vector machine (SVM), random forest, or neural network, is selected to train the data. During training, the model learns the characteristic patterns of normal and anomaly data and establishes classification boundaries or a predictive model. When a new deviant data point is input, the model analyzes its features and calculates the probability that the data point is an anomaly or the degree of deviation from the normal pattern, thereby generating an anomaly score. The anomaly level of the deviant data point is determined based on pre-set anomaly classification criteria (for example, an anomaly score of 0-0.3 is considered mild, 0.3-0.6 is considered moderate, and 0.6-1 is considered severe).

[0109] Finally, the system determines whether a deviating data point is an outlier based on its anomaly level. A threshold is set for this level. For example, data points with a medium anomaly level or higher (anomaly score ≥ 0.3) are considered outliers. If the anomaly level of a deviating data point reaches or exceeds this threshold, it is identified as an outlier and relevant information, such as the data type, specific value, anomaly level, and the triggering judgment rule, is recorded for further analysis and processing. Conversely, if the anomaly level is below the threshold, the deviating data point may be considered to deviate from statistical and business logic perspectives, but after in-depth machine learning analysis, the anomaly is not considered a true outlier. The system may instead flag and archive these deviating data points below the threshold to monitor their changing trends during subsequent data monitoring. If these data points exhibit an increased anomaly level or other abnormal characteristics over time, the system can promptly reassess their status to avoid potential underreporting and continuously optimize the judgment rules and parameter settings of the outlier analysis module.

[0110] This optional embodiment uses general statistical principles, combined with actual business rules, for screening. Furthermore, it leverages the powerful pattern recognition capabilities of machine learning for in-depth analysis, enabling accurate identification of true anomalies. This avoids the misjudgments and omissions that can result from relying solely on a single dimension. For example, some data points may deviate from the normal range statistically but be reasonable from a business perspective (e.g., a sales surge caused by holidays). Alternatively, while seemingly abnormal from a business perspective, they may actually be the beginning of a new business trend. In-depth analysis using machine learning can more accurately determine whether these are true anomalies.

[0111] At the same time, comprehensive consideration of statistics, business logic, and machine learning allows it to adapt to complex and ever-changing business scenarios. Business logic and data distribution vary widely across different enterprises, and this module can be customized based on the specific circumstances of each enterprise. For example, in the financial industry, where business logic places extremely high demands on risk control and data anomalies can result in significant financial losses, stricter anomaly level thresholds and business scope rules can be set. Meanwhile, for innovative internet companies, whose business data fluctuates significantly and whose business models evolve rapidly, machine learning sub-rules can be used to more quickly adapt to new data patterns and business logic changes.

[0112] By assigning anomaly levels to deviant data points, quantitative support is provided for enterprise decision-making. Based on the severity of anomaly levels, enterprise managers can rationally allocate resources for data verification and problem resolution. For example, severely anomalous data points can be prioritized for in-depth investigation by a professional team, who can analyze the cause and implement appropriate corrective measures. Minor anomalies can be observed first, with subsequent data changes contributing to a decision on whether to take action. This quantitative approach helps improve operational efficiency and avoids blindly addressing data anomalies.

[0113] Optionally, the impact path tracing module is specifically configured to:

[0114] Obtaining a business association rule and a data association rule corresponding to the data type of the abnormal point;

[0115] Add the abnormal point as the initial node to the propagation path queue;

[0116] Determining a downstream associated node of the initial node according to the business association rule and the data association rule;

[0117] Adding the downstream associated nodes to the propagation path queue in sequence to obtain a preliminary propagation path;

[0118] The preliminary propagation path is recursively expanded until no further expansion is possible, thereby obtaining the propagation path.

[0119] Specifically, the impact path tracing module first extracts business association rules and data association rules for the data types corresponding to the outliers from the company's business logic documents, data dictionary, data flow diagrams and other materials. Business association rules are formulated based on the company's actual business processes and management requirements, and describe the relationship between different data types at the business level. For example, in financial business, revenue data and cost data jointly affect profit data, that is, profit = revenue - cost, which is a business association rule. Data association rules describe the association relationship between data in database tables, data warehouses and other structures from the perspective of data storage and processing. For example, in a database, the order table and customer table are associated through the customer number field, and the order table and product table are associated through the product number field.

[0120] After obtaining business and data association rules, the impact path tracing module uses the identified outlier as the initial node and places it into the propagation path queue. A propagation path queue is a data structure used to store and manage propagation path nodes, using the first-in, first-out (FIFO) principle. For example, suppose a product's monthly sales data is found to be an outlier in the sales data. The initial node corresponding to this outlier contains information such as the data type (sales amount), data point (specific sales value), and association rules (rules for association with data such as sales volume and price). After adding this node to the propagation path queue, the queue now contains only this initial node. Next, based on the business and data association rules, the downstream associated nodes of the initial node are analyzed. Downstream associated nodes are other data nodes in the business or data processing flow that are affected by the data from the initial node. Based on the above business association rules, profit data is affected by sales data, so the node corresponding to the profit data is the downstream associated node of the sales outlier. From the perspective of data association rules, in a database, if sales data is stored in a sales detail table, and the sales detail table is linked to the sales summary table via date and product number fields, then the node corresponding to the profit data in the sales summary table is the downstream associated node. The module parses these association rules to find the downstream associated nodes of the initial node at the business and data levels, and records the relevant information of these nodes, such as data type, data point, association path, etc.

[0121] After determining the downstream associated nodes of the initial node, these downstream associated nodes are sequentially added to the propagation path queue. Taking the aforementioned sales anomaly as an example, once the downstream associated nodes corresponding to the profit data are added to the queue, the propagation path queue now includes the sales anomaly (initial node) and the profit data node (downstream associated node). At this point, the impact path tracing module checks whether the order of the nodes in the queue aligns with the actual order of the business and data processing processes, adjusting the nodes appropriately to form a preliminary propagation path. For example, according to the business process, sales first influence profit, which then influences other financial indicators (such as net profit). Arranging the nodes in the queue in this order yields a preliminary propagation path starting from the sales anomaly and sequentially passing through the profit data nodes.

[0122] Finally, the impact path tracing module recursively expands the initial propagation path. Recursive expansion involves repeating the above process of identifying and adding downstream associated nodes to the queue, starting with each node in the propagation path queue as a new initial node. Taking the profit data node in the above example as an example, the impact path tracing module, based on business and data association rules, locates downstream associated nodes of the profit data node, such as the tax data node (profit affects tax amounts), and adds these nodes to the propagation path queue. This process continues until no new downstream associated nodes can be found. At this point, the propagation path queue contains all related nodes extending from the initial anomaly point along the business and data association relationships, forming a complete propagation path. For example, the resulting propagation path is: sales anomaly point → profit data node → tax data node → financial statement summary data node, etc., covering the entire potential impact range of the anomaly point in the business and data processing processes.

[0123] This optional embodiment combines business and data association rules, taking into account not only the logical relationships of data at the business level but also the technical relationships involved in data storage and processing. This allows for accurate tracking of all data nodes potentially affected by an anomaly. This avoids missing the full impact of focusing solely on local relationships, providing a more reliable foundation for comprehensive data quality assessments.

[0124] This embodiment starts from the initial outlier point and gradually determines the downstream associated nodes through recursive expansion, which can accurately locate each data node affected by the outlier point. Compared with the traditional impact analysis method based on simple association rules, this method can more carefully sort out the complex relationships between data, especially for multi-level, meshed data association structures, and can accurately identify the affected nodes at each level, thereby improving the accuracy of impact path tracing. By constructing a propagation path queue and processing nodes in a certain order, the impact path tracing process has higher efficiency. The first-in-first-out queue principle ensures the sequential processing of nodes, avoids repeated processing and confusion, and speeds up the generation of propagation paths. When faced with large-scale report data and complex association relationships, this orderly and efficient tracking method can significantly improve the efficiency of data quality assessment and provide timely support for corporate decision-making.

[0125] Optionally, the impact path tracing module is further configured to:

[0126] Determining, based on the business association rules, the node corresponding to the subsequent link of the initial node in the business process;

[0127] Determining, according to the data association rule, nodes corresponding to relevant data points of the initial node in the data structure;

[0128] The node corresponding to the subsequent link and the node corresponding to the relevant data point are used as the downstream associated nodes of the initial node.

[0129] Specifically, the impact path tracing module extracts business association rules from an enterprise's business process documents and business operation specifications. These rules detail the sequence and interrelationships between different business links. For example, in an enterprise's production process, raw material procurement is followed by production and processing, which is followed by quality inspection, and finally, product warehousing. These links have clear business associations. Specifically, the module uses interfaces provided by the enterprise resource planning (ERP) system or business process management (BPM) system to obtain detailed business process information. These systems typically define the various business process links and their flow rules in the form of workflows. By parsing these workflow definitions, the module can identify the business link where the initial node resides and, subsequently, the nodes corresponding to subsequent links. For example, if the initial node is an anomaly in the purchase order data in the raw material procurement link, then according to the business process definition, the corresponding node for the subsequent link is the production plan data node in the production and processing link, as the production plan is affected by the raw material procurement situation.

[0130] After determining the nodes corresponding to the subsequent steps in the business process for the initial node, the module further determines the nodes corresponding to the initial node's related data points in the data structure based on data association rules. Data association rules are primarily derived from the company's database design documents and data dictionary. Taking a company's financial database as an example, in the data structure, the sales order table, customer information table, and product information table are associated via foreign key constraints. The customer number field in the sales order table is associated with the customer number field in the customer information table, and the product number field is associated with the product number field in the product information table. Furthermore, within the sales order table, the sales amount field is associated with the sales quantity field and the product unit price field: sales amount = sales quantity × product unit price. Specifically, the module uses database queries to retrieve the data points associated with the initial node. For example, if the initial node is an outlier in the sales order table, the module can execute a SQL query to retrieve all other data records with the same customer number as the sales order (such as the customer's historical orders, customer credit limit, etc.), as well as data points related to the product number, such as product inventory information and product cost information, to determine the nodes corresponding to these related data points. After determining the nodes corresponding to subsequent links and the nodes corresponding to related data points from the perspectives of business association rules and data association rules respectively, the module unifies these nodes into downstream associated nodes. These downstream associated nodes cover all related nodes that may be affected by the initial node in the two dimensions of business process advancement and data structure association.

[0131] Taking an outlier in sales data as an example, the nodes corresponding to subsequent links include the production plan data node (the subsequent link in the business process), and the nodes corresponding to related data points include customer historical order data nodes, customer credit limit data nodes, product inventory data nodes, product cost data nodes, etc. (data structure association). These downstream related nodes are organized and stored in a data structure to facilitate subsequent recursive expansion processing.

[0132] In this optional embodiment, by combining business association rules with data association rules, the impact of the initial node at both the business process and data structure levels can be fully covered. Business association rules ensure that nodes in subsequent business links that may be affected by the outlier are identified from the perspective of actual business operations; data association rules, from the perspective of data storage and processing, mine nodes corresponding to all data points that are directly or indirectly associated with the initial node. The use of this dual rule avoids the omission of the impact range that may be caused by single-dimensional analysis and improves the integrity of the impact path tracking.

[0133] By determining the nodes corresponding to subsequent links and related data points in the two dimensions of business process and data structure, each downstream related node that may be affected by the initial node can be accurately located. Compared with the method of analyzing only based on business process or data structure, this comprehensive method of the embodiment of the present invention can more accurately identify the affected nodes in complex business and data environments. For example, the potential impact of an abnormal point in sales order data on subsequent production plans can be discovered through business association rules, and its impact on customer credit assessment, inventory management, cost accounting and other aspects can be further discovered through data association rules, thereby providing a reliable basis for subsequent accurate analysis and processing.

[0134] Nodes corresponding to subsequent links and related data points are considered downstream associated nodes, providing a strong guarantee for the accuracy of data quality assessments. During the data quality assessment process, the impact of outliers on the overall quality of report data can be comprehensively assessed from multiple perspectives. By comprehensively considering these downstream associated nodes, the impact of data anomalies on business decisions, financial statements, and operational performance can be more accurately assessed, helping business managers make more scientific and rational decisions, optimizing the company's data management and quality control strategies, and improving operational efficiency and competitiveness.

[0135] Optionally, the affected data analysis module is specifically configured to:

[0136] According to the propagation path, acquiring the data point in the propagation path;

[0137] Performing an impact degree analysis on the data points to obtain an impact degree value of each data point;

[0138] Determining the affected data points according to the affected degree value;

[0139] The affected data points are marked, and a set of affected data points is generated.

[0140] Specifically, the affected data analysis module uses the propagation path generated by the impact path tracing module to obtain data points along the propagation path, which includes a series of data nodes that may be affected by the anomaly. By invoking a data access interface, the module extracts the corresponding data points from the enterprise's database or data warehouse based on the detailed information recorded for each node in the propagation path (such as data type, data point location, and association rules). For example, if the propagation path includes sales data nodes, inventory data nodes, and financial data nodes, the module will extract sales data points from the sales database table, inventory data points from the inventory database table, and financial data points from the financial database table, respectively. These data points will be used as the subjects for subsequent impact analysis. After obtaining the data points along the propagation path, the module performs an impact analysis on each data point.

[0141] In a preferred embodiment of the present invention, the analysis method includes multiple ways to quantify the degree of impact, specifically including:

[0142] Deviation calculation: For each data point, the deviation between the actual value and the expected value is calculated. The expected value can be based on an average of historical data, a trend forecast, or a reasonable value defined by business rules. For example, for an inventory data point, the expected value could be an ideal inventory level calculated based on historical sales data and inventory turnover. The difference between the actual and expected values ​​is the deviation, and the absolute or relative value of the deviation can be used as an indicator of the impact.

[0143] Correlation analysis is used to analyze the strength of the correlation between data points and outliers. By calculating the correlation coefficient (such as the Pearson correlation coefficient or the Spearman correlation coefficient) between the data point and the outlier data, the linear or nonlinear relationship between them can be quantified. The closer the absolute value of the correlation coefficient is to 1, the more likely the data point is affected by the outlier, and the greater the degree of impact. For example, if the correlation coefficient between the outlier in sales data and the profit data point is 0.8, it indicates a strong correlation between the two, and the profit data point is likely to be significantly affected by the sales anomaly.

[0144] In some cases, causal analysis methods (such as Granger causality tests and structural equation models) are used to determine whether a causal relationship exists between data points and outliers. If a causal relationship exists, the strength of the causal relationship is further evaluated to determine the degree of impact. For example, in production data, outliers in raw material quality may causally affect product qualification data points. Granger causality tests can verify the existence of this causal relationship and quantify its degree of impact. Combining the above analysis methods, an impact value is calculated for each data point. This value is a comprehensive score, calculated by weighting multiple factors such as deviation, correlation, and causality. For example, the deviation weight is set to 0.4, the correlation weight is set to 0.3, and the causality weight is set to 0.3. The final impact value is calculated based on the quantified results of each factor.

[0145] Based on the calculated impact value, the module sets an impact threshold to identify affected data points. In this embodiment, the threshold can be set based on historical data analysis, business experience, or statistical principles. For example, a reasonable threshold can be determined by analyzing the distribution of impact values ​​under normal and abnormal conditions in historical data. Assuming the impact threshold is set to 0.5, data points with a calculated impact value greater than or equal to 0.5 are considered affected. Data points with a calculated impact value less than 0.5 are considered less affected and are not considered affected. The affected data points are then labeled with detailed information, including the data point's identifier, impact value, and location on the propagation path. For example, an affected profit data point is labeled as "Affected data point - Profit - Impact value 0.6 - Located in the second link of the propagation path." These labeled affected data points are then collected to generate a set of affected data points. This set can be stored in a list, table, or data structure to facilitate subsequent data evaluation modules in conducting overall quality assessments and generating quality assessment reports.

[0146] In this optional embodiment, the impact value is comprehensively calculated through multiple analytical methods (deviation calculation, correlation analysis, causal relationship inference, etc.), accurately quantifying the degree to which each data point in the propagation path is affected by the outlier. Compared to a single analytical method, this approach in this embodiment more comprehensively and objectively reflects the actual impact on the data, providing a reliable data foundation for subsequent data quality assessment. Determining affected data points based on the impact threshold effectively distinguishes between key data points that are significantly affected by the outlier and those that are less affected. This helps companies focus on and address data issues that truly have a significant impact on business and decision-making, improving the efficiency and relevance of data management. Affected data points are labeled and generated into a collection, providing a clear view of the data impact. This collection provides an intuitive understanding of the scope and extent of the outlier's propagation impact in report data, facilitating the development of reasonable data correction strategies and business response measures.

[0147] Optionally, the data evaluation module is specifically configured to:

[0148] Establish a data quality assessment model;

[0149] Determining the severity of the outlier and each of the affected data points and the impact range of the outlier using the data quality assessment model;

[0150] Performing an overall quality assessment on the report data using the severity value of the outlier and each of the affected data points, as well as the impact range of the outlier;

[0151] Determine the abnormality level of the report data based on the evaluation result, and generate a list of abnormality details of the report data;

[0152] The quality assessment report is generated based on the abnormality details list and the quality score.

[0153] Specifically, historical report data is collected and organized, encompassing multiple dimensions such as accuracy, completeness, consistency, and timeliness. A database containing a large number of data samples is constructed to train and optimize data quality assessment models. Machine learning algorithms, such as decision trees and neural networks, are used to learn and analyze these samples, enabling the model to identify patterns and regularities in data quality.

[0154] In a preferred embodiment of the present invention, before applying a machine learning algorithm, the collected report data samples are preprocessed. Data preprocessing includes steps such as data cleaning, data integration, data conversion, and data reduction. Data cleaning is used to remove noise and address missing values; data integration combines data from different data sources; data conversion, such as normalization and standardization, converts data into a format suitable for algorithm processing; and data reduction is used to reduce data volume and improve algorithm efficiency. Preprocessing ensures data quality and consistency, providing a reliable data foundation for subsequent machine learning algorithm applications.

[0155] Feature selection and engineering are performed on preprocessed data to extract key features related to data quality. These features may include data accuracy indicators (such as the deviation between data and actual values), completeness indicators (such as the proportion of missing values), consistency indicators (such as the degree of inconsistency between data), and timeliness indicators (such as the frequency of data updates). Based on business needs and data characteristics, statistical analysis and correlation analysis are used to select the features most influential in data quality assessment. Feature combinations and construction are then performed to improve model performance and generalization capabilities.

[0156] Use machine learning algorithms, such as decision trees and neural networks, to study and analyze processed data samples and train data quality assessment models.

[0157] In one embodiment, the selected machine learning algorithm is a decision tree, a supervised learning algorithm based on a tree structure. During training, the dataset is continuously divided into subsets by selecting appropriate features and split points until a stopping condition is met. For data quality assessment, the decision tree can determine whether a data point is abnormal data and determine its severity based on the values ​​of the features. For example, a decision tree model can be constructed based on features such as data accuracy and completeness as the splitting criteria. Each internal node represents a test of a feature, each branch represents the test result, and each leaf node represents a category (such as normal data, mildly abnormal data, severely abnormal data, etc.) or outputs a continuous value (such as a severity value). The decision tree is trained using the training dataset, and the tree structure and parameters are adjusted so that the model can accurately identify patterns and regularities in data quality.

[0158] In another embodiment, the selected machine learning algorithm is a neural network, an algorithm that mimics the neural network structure of the human brain. A neural network is composed of a large number of neurons (nodes) that transmit and process information through connection weights. In data quality assessment, a feature vector is used as the input of the neural network. After undergoing nonlinear transformations through multiple layers of neurons, the data severity value or quality category is ultimately output. For example, a multi-layer perceptron (MLP) neural network model is constructed, comprising an input layer, hidden layers, and an output layer. The input layer receives data quality-related features, the hidden layer extracts and combines features using activation functions, and the output layer provides the assessment results. Using a backpropagation algorithm, the neural network weights are iteratively updated using the training dataset to optimize the model's performance and enable it to learn the complex patterns and laws of data quality.

[0159] During model training, we use methods such as cross-validation to verify the model and evaluate its accuracy and generalization ability. By adjusting the algorithm parameters, such as the depth of the decision tree, the number of layers and neurons in the neural network, we can avoid overfitting or underfitting the model and improve the model's prediction effect.

[0160] When report data is input into the data quality assessment model, the model uses its learned anomaly feature recognition rules to quickly locate outliers within the data. For each outlier, the model calculates its deviation from normal data and uses standardized formulas, such as statistically based standard deviation or mean square error, to determine its severity. This quantifies the outlier's impact on data quality. Furthermore, the model leverages data correlations and propagation paths to analyze the scope of the outlier's impact on surrounding data points. For example, for an outlier in a financial statement, by analyzing its correlations with other data points, such as costs and profits, the model can determine the scope of the outlier's impact, including directly related cost and profit accounting data points, as well as the impact level and scope within the entire report data. This process accurately identifies which data points are affected by the outlier, providing an accurate basis for further quality assessment. By comprehensively considering the outlier, the severity of each affected data point, and the scope of the outlier's impact, a weighted summation or other mathematical methods are used to conduct an overall quality assessment of the report data. Based on the preset evaluation model algorithm, the severity of each outlier is multiplied by its corresponding impact weight, and then added to the quality weight of other normal data points to produce a comprehensive quality score. For example, for a report containing multiple outliers, the overall quality penalty for each outlier is calculated based on its severity and impact, and then these penalty points are subtracted from the total score to obtain the overall quality score. A detailed quality assessment report is also generated, listing issues within the report data, the location of the outliers, and the impact range, allowing users to gain a comprehensive understanding of data quality.

[0161] Based on the results of the overall quality assessment, further data analysis and statistics are used to calculate the severity of report data anomalies. This is achieved by summarizing and normalizing the severity values ​​of all anomalies, providing a visual representation of the degree of anomaly within the report data. Furthermore, a detailed list of anomaly details is generated based on the detailed information for each anomaly, including anomaly type, location, severity, and impact range, following specific sorting rules and formatting requirements. This list clearly displays each anomaly in the report in a table or list format, allowing users to quickly locate and view detailed information about the anomaly data, providing a strong basis for subsequent data processing and correction.

[0162] Combine the anomaly details list and quality score with a report generation template and formatting tools to generate a complete and detailed data quality assessment report. This report includes an overview of the data quality assessment, assessment methods and models, anomaly analysis, an overall quality score, and improvement suggestions. The anomaly analysis section details each anomaly, while the overall quality score presents the overall data quality level in intuitive charts or numerical values. The improvement suggestions section provides users with reference suggestions based on the assessment results. By generating a quality assessment report, users can gain a comprehensive understanding of the quality of their report data, promptly identify and resolve data quality issues, and improve data reliability and availability.

[0163] The report includes:

[0164] Data Quality Overview: A brief description of the overall quality of the report data, including key indicators such as data quality score, quality level, and abnormality level.

[0165] Anomaly Details Analysis: Detailed analysis of each anomaly point and affected data point in the anomaly details list, explaining the specific circumstances and possible causes of the anomaly or impact. For example, if a sales data anomaly is significantly higher than the same period in history, this could be due to a sudden change in the market environment or a data entry error. The affected profit data point may have a significant deviation in the calculated result due to the anomaly in the sales data.

[0166] Impact Scope Assessment: Describe the impact scope of the anomaly, including the business segments and data types involved, and the potential impact on other businesses and data. For example, sales data anomalies not only affect profit calculations, but also inventory management, production planning, and other business segments.

[0167] Improvement Suggestions: We propose corresponding improvement suggestions for discovered data quality issues, such as strengthening data entry review, optimizing data monitoring rules, and conducting root cause analysis of data anomalies, to help companies improve their data quality management.

[0168] In this optional embodiment, by establishing a data quality assessment model, anomalies in report data can be quickly and accurately identified, avoiding the tediousness and error-proneness of manual inspection, and greatly improving the efficiency and accuracy of data anomaly identification. The severity values ​​and impact range of anomalies and affected data points are determined, and an overall quality assessment of the report data is performed, achieving a comprehensive quantitative analysis of data quality. This makes the data quality assessment results more objective and accurate, providing reliable data support for data management and decision-making. The generated anomaly details list and quality assessment report provide users with detailed abnormal data information and improvement suggestions, helping users to promptly identify data problems and take effective improvement measures, improving the quality and credibility of report data, and thereby improving the company's data governance level and decision-making efficiency. Through a strict data quality assessment and supervision mechanism, the quality of report data is effectively guaranteed, reducing system failures and business risks caused by data quality issues, and enhancing the reliability and stability of the entire system.

[0169] Combine Figure 2 As shown, the present invention also provides a report data anomaly monitoring and quality assessment method, comprising:

[0170] Obtaining enterprise report data, and analyzing the report data to obtain data points corresponding to each data type in the report data;

[0171] Analyze the data points of the data type, and determine whether there are abnormal points in the data points in combination with a preset judgment rule corresponding to the data type;

[0172] When the abnormal point exists in the data points, the path of the abnormal point is tracked by a path analysis algorithm to determine the propagation path corresponding to the abnormal point;

[0173] Performing an impact analysis on the data points in the propagation path to determine the affected data points;

[0174] The overall quality level of the report data is evaluated based on the affected data points in combination with the abnormal points; and a quality evaluation report of the report data is generated based on the overall quality level.

[0175] The advantages of the report data anomaly monitoring and quality assessment method of the present invention over the prior art are the same as the advantages of the above-mentioned report data anomaly monitoring and quality assessment system over the prior art, and will not be repeated here.

[0176] Combine Figure 3 As shown, the present invention also provides an electronic device, including a memory and a processor;

[0177] The memory is used to store computer programs;

[0178] The processor is used to implement the above-mentioned report data anomaly monitoring and quality assessment method when executing the computer program.

[0179] The advantages of the electronic device of the present invention over the prior art are the same as the advantages of the above-mentioned report data anomaly monitoring and quality assessment system over the prior art, and will not be repeated here.

[0180] Although the present invention is disclosed as above, the scope of protection disclosed by the present invention is not limited thereto. Those skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention, and these changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A report data anomaly monitoring and quality assessment system, characterized in that: include: A data acquisition module is used to acquire the enterprise's report data and analyze the report data to obtain data points corresponding to each data type in the report data; An outlier analysis module, configured to analyze the data points of the data type and determine whether there are outliers in the data points in combination with a preset judgment rule corresponding to the data type; An impact path tracking module is used to, when the abnormal point exists in the data points, track the path of the abnormal point by using a path analysis algorithm to determine the propagation path corresponding to the abnormal point; An affected data analysis module, configured to analyze the degree of impact of the data points in the propagation path and determine the affected data points; The affected data analysis module is specifically used to: According to the propagation path, acquiring the data point in the propagation path; Performing an impact degree analysis on the data points to obtain an impact degree value of each data point; Determining the affected data points according to the affected degree value; Marking the affected data points and generating a set of affected data points; a data evaluation module, configured to evaluate the overall quality level of the report data based on the affected data points in combination with the abnormal points, and generate a quality evaluation report for the report data based on the overall quality level; The data evaluation module is specifically used to: Establish a data quality assessment model; Determining the severity of the outlier and each of the affected data points and the impact range of the outlier using the data quality assessment model; Performing an overall quality assessment on the report data using the severity value of the outlier and each of the affected data points, as well as the impact range of the outlier; Determine the abnormality level of the report data based on the evaluation result, and generate a list of abnormality details of the report data; The quality assessment report is generated based on the abnormality details list and the quality score.

2. The report data anomaly monitoring and quality assessment system according to claim 1, characterized in that: The data acquisition module is specifically used to: Obtaining initial report data from the database of the enterprise; Performing format conversion and preprocessing on the initial report data to obtain standardized report data; Identify the data according to the metadata information of the report data to obtain multiple data types in the report data; The data points corresponding to the data type are extracted according to the data type.

3. The report data anomaly monitoring and quality assessment system according to claim 1, characterized in that: The outlier analysis module is specifically used to: Determining the preset judgment rule according to the data type, wherein the preset judgment rule includes a statistical sub-rule, a business logic sub-rule, and a machine learning sub-rule; screening the data points of the data type according to the statistical sub-rules, the business logic sub-rules, and the machine learning sub-rules in sequence; Determine whether there are abnormal points in the data points of the data type according to the screening result.

4. The report data anomaly monitoring and quality assessment system according to claim 3, characterized in that: The outlier analysis module is further configured to: Determining, based on the statistical sub-rules, whether there are any deviating data points in the data points that exceed a preset normal distribution; If the deviated data point exists in the data points, determining whether the deviated data point exceeds the business scope corresponding to the data type according to the business logic sub-rule; If the deviated data point exceeds the business scope, performing an in-depth analysis on the deviated data point according to the machine learning sub-rule to obtain an abnormality level of the deviated data point; According to the abnormality level, it is determined whether the deviated data point is the abnormal point.

5. The report data anomaly monitoring and quality assessment system according to claim 1, characterized in that: The impact path tracing module is specifically used to: Obtaining a business association rule and a data association rule corresponding to the data type of the abnormal point; Add the abnormal point as the initial node to the propagation path queue; Determining a downstream associated node of the initial node according to the business association rule and the data association rule; Adding the downstream associated nodes to the propagation path queue in sequence to obtain a preliminary propagation path; The preliminary propagation path is recursively expanded until no further expansion is possible, thereby obtaining the propagation path.

6. The report data anomaly monitoring and quality assessment system according to claim 5, characterized in that: The impact path tracing module is further configured to: Determining, based on the business association rules, the node corresponding to the subsequent link of the initial node in the business process; Determining, according to the data association rule, nodes corresponding to relevant data points of the initial node in the data structure; The node corresponding to the subsequent link and the node corresponding to the relevant data point are used as the downstream associated nodes of the initial node.

7. A method for monitoring and assessing report data anomalies, characterized in that: include: Obtaining enterprise report data, and analyzing the report data to obtain data points corresponding to each data type in the report data; Analyze the data points of the data type, and determine whether there are abnormal points in the data points in combination with a preset judgment rule corresponding to the data type; When the abnormal point exists in the data points, the path of the abnormal point is tracked by a path analysis algorithm to determine the propagation path corresponding to the abnormal point; Performing an impact analysis on the data points in the propagation path to determine the affected data points; specifically comprising: obtaining the data points in the propagation path according to the propagation path; Performing an impact degree analysis on the data points to obtain an impact degree value of each data point; Determining the affected data points according to the affected degree value; Marking the affected data points and generating a set of affected data points; evaluating the overall quality level of the report data based on the affected data points and the abnormal points; and generating a quality evaluation report for the report data based on the overall quality level; Specifically include: establishing a data quality assessment model; Determining the severity of the outlier and each of the affected data points and the impact range of the outlier using the data quality assessment model; Performing an overall quality assessment on the report data using the severity value of the outlier and each of the affected data points, as well as the impact range of the outlier; Determine the abnormality level of the report data based on the evaluation result, and generate a list of abnormality details of the report data; The quality assessment report is generated based on the abnormality details list and the quality score.

8. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is configured to implement the report data anomaly monitoring and quality assessment method according to claim 7 when executing the computer program.

Images