Supervision data quality monitoring system and method based on artificial intelligence

Abstract

The present invention application discloses a regulatory data quality monitoring system and method based on artificial intelligence, which relates to the field of artificial intelligence technology. The present invention application sets a fixed collection cycle, standardizes the acquisition of business interface call data and constructs a dual index set to ensure timely and standardized data collection, integrates scattered information, and provides a high-quality foundation for subsequent analysis, thereby improving efficiency and accuracy. With the help of weighted fusion calculations and time series prediction models, the prediction threshold is calculated based on peak data, and the number of historical data prediction calls and anomalies is combined to build a multi-level monitoring system to accurately capture dynamic changes, proactively analyze trends, and improve the real-time nature of anomaly detection and the accuracy of early warning. Based on the system load and anomaly trend graph, the three sigma principle is used to set dynamic monitoring thresholds, and the predicted values ​​are compared to achieve graded early warnings, changing the limitations of traditional fixed thresholds, accurately capturing abnormal fluctuations, and timely warnings, providing a quantitative basis for dynamic management and control of data quality.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and specifically to an artificial intelligence-based regulatory data quality monitoring system and method. Background Art

[0002] In digital regulatory scenarios, data quality is a core element in ensuring the scientific nature of regulatory decisions and the stability of system operations. Accurate and reliable data quality testing can promptly identify anomalies in business interface calls and effectively mitigate regulatory loopholes and decision-making errors caused by data bias. This is crucial for improving regulatory efficiency, reducing operating costs, and enhancing the system's resilience to risk. It is also a necessary prerequisite for achieving digital transformation and sustainable development in regulation.

[0003] In the current field of regulatory data quality monitoring, traditional methods struggle to meet the demands of high-frequency, dynamically changing business interface call scenarios. Particularly in large-scale regulatory systems, traditional methods lack the ability to dynamically analyze business interface load, resulting in insufficient real-time performance and delayed anomaly detection. These methods typically employ fixed thresholds or static models for monitoring, which are unable to adapt to dynamic changes in business interface call frequency and anomaly ratios, severely impacting the stability of regulatory systems and data reliability. Therefore, a monitoring method that can dynamically analyze business interface load and accurately predict call trends and anomalies is urgently needed to improve data quality detection. Summary of the Invention

[0004] The purpose of this invention application is to provide an artificial intelligence-based regulatory data quality monitoring system and method to solve the problems raised in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solutions: a method for monitoring the quality of regulatory data based on artificial intelligence, wherein the intelligent management method comprises the following steps:

[0006] Step S1: Set a supervision data collection cycle, obtain call data of all business interfaces according to the supervision data collection cycle, and build a business interface load analysis set;

[0007] Step S1-1, setting a supervision data collection period T, wherein the supervision data collection period T is a fixed time interval;

[0008] Step S1-2: Based on the set regulatory data collection cycle T, obtain call data of all business interfaces according to the regulatory data collection cycle T through application program interface (API) calls or log analysis;

[0009] Step S1-3: standardize the acquired business interface call data, including data cleaning and data format unification;

[0010] Step S1-4: construct a business interface load analysis set using the standardized business interface call data with the timestamp as the primary index and the unique identifier of the business interface as the secondary index.

[0011] By setting a fixed collection cycle, orderly acquiring and standardizing business interface call data, and constructing a business interface load analysis set in a dual-index manner, we can ensure the timeliness and standardization of data collection, effectively integrate scattered call information, and provide a high-quality data foundation with a clear structure and unified format for subsequent data statistics, trend analysis, and quality monitoring, thereby improving analysis efficiency and accuracy.

[0012] Step S2: Analyze the service interface load analysis set to obtain the call count and call exception count data corresponding to the service interface timestamp, obtain the service interface call data for each regulatory data collection period in history, and analyze it to obtain the peak call count of the service interface;

[0013] Step S2-1: Based on the service interface load analysis set, group and count the service interface call data by timestamp and service interface unique identifier;

[0014] Step S2-2: Analyze and extract the number of calls and call exceptions for each timestamp of the business interface;

[0015] Step S2-3: Obtain the call data of the business interface in each supervision data collection period in the history of the business interface, obtain the maximum number of calls of the business interface through bubble sort, and record it as the peak number of calls.

[0016] The data of the business interface load analysis set is grouped and counted, and the number of business interface calls and exceptions at each timestamp is extracted. The peak number of calls is obtained by bubble sorting based on historical data. This effectively analyzes the fluctuation characteristics of the business interface, quantifies the intensity and exceptions of business interface calls, and identifies historical call extremes. This provides key data support for subsequent call trend prediction, system load assessment, and exception warning, thereby enhancing the pertinence and reliability of data monitoring.

[0017] Step S3: Analyze and predict the peak call count of the service interface to obtain the call count trend of the service interface, and analyze the call exception ratio trend based on the call exception data of the service interface corresponding timestamp; construct a call exception change trend chart based on the call exception data of the service interface, and combine the call exception trend analysis and prediction to obtain the call exception number of the service interface;

[0018] Step S3-1: Select the timestamp corresponding to the peak call count as the starting point, obtain the call count data between the service interface starting point and the current time period that is only less than the peak call count, and record it as the second peak call count;

[0019] Step S3-2: Based on the peak call count and the second peak call count of the business interface, calculate the maximum call count of the business interface in the next regulatory data collection period T through weighted fusion calculation, and record it as the call count prediction threshold;

[0020] The formula for calculating the maximum number of calls within the next regulatory data collection cycle T of the business interface obtained by weighted fusion calculation is as follows:

[0021] ;

[0022] Where, F max，new It represents the maximum number of calls within the next regulatory data collection cycle T of the business interface; w represents the weight coefficient of the peak number of calls; F max，1 Indicates the peak number of calls to the business interface; F max，2 Represents the call count data between the starting point of the business interface and the current time period, which is just less than the peak call count.

[0023] Step S3-3: Obtain the call count data for each regulatory data collection cycle T of the business interface history, select a call count prediction threshold as the upper limit of the predicted value of the business interface call count, and use the exponential smoothing method in the time series prediction model to predict the call count of the business interface in the next regulatory data collection cycle T, which is recorded as the call count prediction value;

[0024] Exponential smoothing forecast calculations use the following formula:

[0025] ;

[0026] Where, F T+1 It is represented by the predicted number of calls of the business interface in the next regulatory data collection cycle T; a is represented by the average coefficient, which ranges from 1 to 0; Y T It is represented by the actual number of calls in the current regulatory data collection period T; F T It is represented as the predicted value of the number of calls in the current regulatory data collection cycle T;

[0027] By applying exponential smoothing within a time series forecasting model, combined with historical call count data for each regulatory data collection cycle, and using the call count prediction threshold as the upper limit for the predicted value, we achieve a dynamic forecast of the number of calls to the business interface during the next regulatory data collection cycle. Furthermore, the average coefficient in the time series forecasting model is adaptively adjusted based on data fluctuations, enabling the model to learn call patterns and trends from historical data, enabling it to automatically identify data features similar to artificial intelligence.

[0028] When new data is accessed, the time series prediction model will weight the current data based on historical learning results, and then deduce and predict the number of future calls. This data-driven dynamic parameter adjustment and pattern recognition logic fully reflects the core characteristics of artificial intelligence technology in the field of data prediction: self-optimization and continuous evolution, which is significantly different from traditional fixed parameter models.

[0029] Step S3-4: Perform primary monitoring of the business interface based on the call count prediction threshold. The primary monitoring and early warning judgment is as follows:

[0030] Step S3-4-1: When the predicted number of calls exceeds the predicted number of calls threshold, a business interface warning signal is issued;

[0031] Step S3-4-2: When the predicted number of calls does not exceed the predicted number of calls threshold, continue monitoring the business interface;

[0032] Step S3-5: Divide the number of call exceptions in each supervision data collection period T of the business interface history by the number of calls in the corresponding supervision data collection period T to obtain a call exception ratio;

[0033] Step S3-6: Calculate the average abnormal call count ratio based on the abnormal call count ratio of each historical regulatory data collection period T of the business interface;

[0034] Step S3-7: multiply the predicted number of calls of the service interface by the average ratio of abnormal calls to obtain the abnormal number of calls of the service interface, which is recorded as the predicted number of abnormal calls.

[0035] Through weighted fusion calculation and combined with the time series prediction model, it is possible to accurately capture the dynamic change characteristics of business interface calls, scientifically set prediction thresholds and generate call count prediction values, and realize forward-looking analysis of business interface call trends; at the same time, based on the average level of historical anomaly ratio calculation, the anomaly count prediction value is generated in combination with the call count prediction value, and an anomaly change trend graph is constructed to form a multi-level anomaly monitoring system from a single business interface to the entire system, effectively improving the real-time performance of anomaly detection and the accuracy of early warning, and providing a data-driven decision-making basis for system load assessment and quality control. With the help of historical data, this method uses the time series prediction model to automatically identify and learn the changing pattern of the business interface call count, and then deduce and predict future data, which is deeply consistent with the application logic and technical characteristics of artificial intelligence in the field of data prediction.

[0036] Step S4: Obtain the total number of system calls and total number of call exceptions based on the call number trend and the abnormality ratio trend of each business interface; obtain the system's historical total number of calls and total number of call exceptions to analyze the load trend and abnormality trend of the business interface;

[0037] Step S4-1: Using the unique identifier of the service interface as an index, read the predicted number of calls and the predicted number of abnormal calls of each service interface in the system;

[0038] Step S4-2: accumulating the predicted call times of each business interface in the system to obtain total call times data; accumulating the predicted call times of each business interface in the system to obtain total call times prediction data;

[0039] Step S4-3: Sort the total number of calls and the total number of abnormal calls in chronological order according to the supervision data collection period T;

[0040] Step S4-3-1: Based on the total call count data of each regulatory data collection period T, construct the x-axis with the regulatory data collection period T as the horizontal axis and the total call count data as the vertical axis to construct the y-axis, thereby constructing a load trend chart of the business interface;

[0041] Step S4-3-2: Based on the total number of abnormal calls in each regulatory data collection period T, the x-axis is constructed with the regulatory data collection period T as the horizontal axis, and the total number of abnormal calls as the vertical axis to construct the y-axis, thereby constructing an abnormal trend graph of the business interface.

[0042] Using the unique identifier of the business interface as the index, the calls and abnormal prediction values ​​of each business interface are summarized and accumulated to form the system total data. After sorting in chronological order, the load trend chart and the abnormal trend chart are constructed respectively. It can integrate the scattered business interface data from the macro level of the system, and intuitively present the overall call load changes and abnormal development trends of the supervision system, effectively making up for the limitations of single business interface analysis, and providing visual and systematic data support for fully understanding the system operation status, identifying potential risks, and formulating dynamic monitoring strategies.

[0043] Step S5: Setting monitoring thresholds based on the system's load trend and abnormal trend, and using the monitoring thresholds to perform data quality detection on the system;

[0044] Based on the load trend chart and the exception trend chart, the average and variance of the number of service interface calls and the number of call exceptions are calculated. The call count monitoring threshold and the exception count monitoring threshold are set based on the upper limit value according to the Three Sigma principle. The calculation formula for setting the call count monitoring threshold is as follows:

[0045] ;

[0046] Where, E A Indicates the call count monitoring threshold; A avgIt is represented by the average number of actual calls to the business interface; n is the number of regulatory data collection cycles T involved in the calculation of the average value; k is the standard deviation multiplier, which is determined by the Three Sigma principle; a i It is expressed as the number of calls during the i-th regulatory data collection cycle T;

[0047] The standard deviation multiplier k is calculated using the following formula:

[0048] ;

[0049] Where, k represents the standard deviation multiple coefficient in the three sigma principle; A max Indicates the upper limit of the number of business interface calls; A min Indicates the lower limit of the number of business interface calls;

[0050] The calculation formula for setting the abnormality number monitoring threshold is as follows:

[0051] ;

[0052] Where, E B Indicates the monitoring threshold of abnormal number; B avg It is represented by the average number of abnormal business interface calls; b i It is represented by the number of abnormal calls during the i-th regulatory data collection period T;

[0053] The system performs data quality detection based on the call count monitoring threshold and the exception count monitoring threshold. The specific process of data quality detection is as follows:

[0054] When the predicted value of the number of calls exceeds the call number monitoring threshold, a first quality warning signal is issued;

[0055] When the predicted number of calls does not exceed the call number monitoring threshold, the system's business interface calls are continuously monitored;

[0056] When the predicted value of the number of call exceptions exceeds the abnormality monitoring threshold, a second quality warning signal is issued;

[0057] When the predicted value of the number of abnormal calls does not exceed the abnormal number monitoring threshold, the system's business interface calls are continuously monitored.

[0058] Based on the system load and anomaly trend chart, by calculating the average and variance of the number of business interface calls and the number of anomalies, the three sigma principle is used to scientifically set the dynamic monitoring threshold, and the predicted value of the number of calls and the predicted value of the number of anomalies are compared with the corresponding thresholds respectively, to achieve hierarchical early warning and continuous monitoring, effectively avoiding the lag and misjudgment risk of traditional fixed threshold monitoring, and can accurately capture abnormal fluctuations in system data quality, trigger the early warning mechanism in time, and provide a quantitative and reliable judgment basis for the dynamic control of regulatory system data quality.

[0059] Furthermore, a supervision data quality monitoring system based on artificial intelligence includes a data acquisition and processing module, a data statistical analysis module, a trend prediction and analysis module, a system data integration module and a quality detection and early warning module;

[0060] The data acquisition and processing module is used to set the supervision data acquisition cycle and obtain and standardize the business interface call data; the data statistical analysis module is used to analyze the business interface load data and obtain the number of calls, abnormalities and peak calls of the business interface; the trend prediction and analysis module is used to predict the trend of the number of calls and abnormalities of the business interface; the system data integration module is used to summarize the predicted data of each business interface and construct a load and abnormality trend chart; the quality detection and early warning module is used to set monitoring thresholds and detect and warn the system data quality based on the thresholds;

[0061] The output end of the data acquisition and processing module is electrically connected to the input end of the data statistical analysis module; the output end of the data statistical analysis module is electrically connected to the input end of the trend prediction and analysis module; the output end of the trend prediction and analysis module is electrically connected to the input end of the system data integration module; the output end of the system data integration module is electrically connected to the input end of the quality detection and early warning module;

[0062] The data collection and processing module includes a collection cycle setting unit and a business interface data processing unit; the collection cycle setting unit is used to set a fixed time interval for the supervision data collection cycle; the business interface data processing unit is used to perform a standardized processing flow for cleaning and formatting the acquired business interface call data;

[0063] The data statistics analysis module includes a call data extraction unit and a peak data acquisition unit; the call data extraction unit is used to group and count the number of calls and exceptions of the service interface according to the timestamp and service interface identifier; the peak data acquisition unit is used to obtain the peak number of calls of the service interface by sorting historical data;

[0064] The trend prediction and analysis module includes a call count prediction unit and an exception count prediction unit; the call count prediction unit is used to predict the call count of the service interface based on peak data and a time series model; the exception count prediction unit is used to estimate the exception count of the service interface based on the historical exception ratio and the call count prediction value;

[0065] The system data integration module includes a forecast data aggregation unit and a trend graph construction unit; the forecast data aggregation unit is used to accumulate the call count and exception count prediction values ​​of each business interface to obtain total data; the trend graph construction unit is used to construct a trend graph with the collection period as the horizontal axis and the total data as the vertical axis;

[0066] The quality detection and early warning module includes a monitoring threshold setting unit and a quality early warning management unit; the monitoring threshold setting unit is used to calculate and set the monitoring threshold according to the statistical characteristics of the trend graph through the three sigma principle; the quality early warning management unit is used to compare the predicted value with the threshold and trigger an early warning signal when the threshold is exceeded.

[0067] Compared with the prior art, the beneficial effects of the present invention are:

[0068] 1. This application establishes a fixed regulatory data collection cycle, systematically acquires and standardizes business interface call data, and constructs a dual-indexed business interface load analysis set using timestamps and unique business interface identifiers. This ensures timely and standardized data collection and effectively integrates dispersed information. Compared to traditional methods, this approach provides a clear, uniformly formatted, high-quality data foundation for subsequent data statistics, trend analysis, and quality monitoring, significantly improving analytical efficiency and accuracy.

[0069] 2. This invention combines weighted fusion calculations with time series prediction models to accurately capture the dynamic characteristics of business interface calls. Prediction thresholds are calculated based on peak and second-peak call counts, and historical data is used to predict call counts and anomaly counts, building a multi-level anomaly monitoring system. Compared to traditional static monitoring, this method proactively analyzes call trends, significantly improving the real-time nature of anomaly detection and the accuracy of early warnings.

[0070] 3. This invention applies the Three Sigma principle to scientifically set dynamic monitoring thresholds based on system load and abnormal trend charts, overcoming the limitations of traditional fixed thresholds. Compared with traditional fixed threshold or static model monitoring methods, this method has significant intelligent advantages. By comparing predicted values ​​with dynamic thresholds to achieve graded early warning and continuous monitoring, it can accurately capture abnormal fluctuations in data quality, trigger early warnings in a timely manner, effectively avoid lags and misjudgment risks, and provide a quantitative and reliable judgment basis for the dynamic management and control of data quality in regulatory systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A flowchart of an artificial intelligence-based regulatory data quality monitoring method is provided for the present invention;

[0072] Figure 2 The present invention applies for a structural diagram of an artificial intelligence-based regulatory data quality monitoring system. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0074] Example 1: Figure 1 As shown, the present invention provides a technical solution, a method for monitoring the quality of regulatory data based on artificial intelligence, and an intelligent management method comprising the following steps:

[0075] Step S1: Set a supervision data collection cycle, obtain call data of all business interfaces according to the supervision data collection cycle, and build a business interface load analysis set;

[0076] Step S1-1, setting a supervision data collection period T, wherein the supervision data collection period T is a fixed time interval;

[0077] Step S1-2: Based on the set regulatory data collection cycle T, obtain call data of all business interfaces according to the regulatory data collection cycle T through application program interface (API) calls or log analysis;

[0078] Step S1-3: standardize the acquired business interface call data, including data cleaning and data format unification;

[0079] Step S1-4: construct a business interface load analysis set using the standardized business interface call data with the timestamp as the primary index and the unique identifier of the business interface as the secondary index.

[0080] In specific implementation, for example, the regulatory data collection cycle T is set to 1 hour, and the API interface is used to pull or parse system logs in real time to obtain the call data of each business interface, such as the call time and return status of the user authentication interface and data query interface. Data collection needs to ensure stability to avoid data loss due to network fluctuations. At the same time, during standardization processing, duplicate records must be eliminated and missing fields must be supplemented to ensure the uniqueness of the dual-index structure timestamp + interface ID.

[0081] Step S2: Analyze the service interface load analysis set to obtain the call count and call exception count data corresponding to the service interface timestamp, obtain the service interface call data for each regulatory data collection period in history, and analyze it to obtain the peak call count of the service interface;

[0082] Step S2-1: Based on the service interface load analysis set, group and count the service interface call data by timestamp and service interface unique identifier;

[0083] Step S2-2: Analyze and extract the number of calls and call exceptions for each timestamp of the business interface;

[0084] Step S2-3: Obtain the call data of the business interface in each supervision data collection period in the history of the business interface, obtain the maximum number of calls of the business interface through bubble sort, and record it as the peak number of calls.

[0085] In practice, database grouping queries, such as SQL's GROUP BY, are used to count call records by timestamp and business interface ID, extracting successful calls and exception return counts, such as records with HTTP status codes other than 200. When using bubble sort to process historical data, consider efficiency issues with large data volumes. Quick sorting or using database indexes to directly retrieve the maximum value can be preferred to ensure the accuracy of peak call counts.

[0086] Step S3: Analyze and predict the peak call count of the service interface to obtain the call count trend of the service interface, and analyze the call exception ratio trend based on the call exception data of the service interface corresponding timestamp; construct a call exception change trend chart based on the call exception data of the service interface, and combine the call exception trend analysis and prediction to obtain the call exception number of the service interface;

[0087] Step S3-1: Select the timestamp corresponding to the peak call count as the starting point, obtain the call count data between the service interface starting point and the current time period that is only less than the peak call count, and record it as the second peak call count;

[0088] Step S3-2: Based on the peak call count and the second peak call count of the business interface, calculate the maximum call count of the business interface in the next regulatory data collection period T through weighted fusion calculation, and record it as the call count prediction threshold;

[0089] Step S3-3: Obtain the call count data for each regulatory data collection cycle T of the business interface history, select a call count prediction threshold as the upper limit of the predicted value of the business interface call count, and use the exponential smoothing method in the time series prediction model to predict the call count of the business interface in the next regulatory data collection cycle T, which is recorded as the call count prediction value;

[0090] Step S3-4: Perform primary monitoring of the business interface based on the call count prediction threshold. The primary monitoring and early warning judgment is as follows:

[0091] Step S3-4-1: When the predicted number of calls exceeds the predicted number of calls threshold, a business interface warning signal is issued;

[0092] Step S3-4-2: When the predicted number of calls does not exceed the predicted number of calls threshold, continue monitoring the business interface;

[0093] Step S3-5: Divide the number of call exceptions in each supervision data collection period T of the business interface history by the number of calls in the corresponding supervision data collection period T to obtain a call exception ratio;

[0094] Step S3-6: Calculate the average abnormal call count ratio based on the abnormal call count ratio of each historical regulatory data collection period T of the business interface;

[0095] Step S3-7: multiply the predicted number of calls of the service interface by the average ratio of abnormal calls to obtain the abnormal number of calls of the service interface, which is recorded as the predicted number of abnormal calls.

[0096] In practice, the weighted fusion calculation can be implemented with a weight coefficient w of 0.6 to increase the peak value. For example, if the peak call count is 1000 and the secondary peak is 800, the prediction threshold is 1000 × 0.6 + 800 × 0.4 = 920. The time series model can use simple exponential smoothing with an α of 0.3, combining call data from the past 10 cycles to predict the value for the next cycle. Model parameters need to be dynamically adjusted based on business fluctuations. For example, in high-frequency trading scenarios, the α value can be reduced to increase the weight of recent data and avoid prediction lags.

[0097] Step S4: Obtain the total number of system calls and total number of call exceptions based on the call number trend and the abnormality ratio trend of each business interface; obtain the system's historical total number of calls and total number of call exceptions to analyze the load trend and abnormality trend of the business interface;

[0098] Step S4-1: Using the unique identifier of the service interface as an index, read the predicted number of calls and the predicted number of abnormal calls of each service interface in the system;

[0099] Step S4-2: accumulating the predicted call times of each business interface in the system to obtain total call times data; accumulating the predicted call times of each business interface in the system to obtain total call times prediction data;

[0100] Step S4-3: Sort the total number of calls and the total number of abnormal calls in chronological order according to the supervision data collection period T;

[0101] Step S4-3-1: Based on the total call count data of each regulatory data collection period T, construct the x-axis with the regulatory data collection period T as the horizontal axis and the total call count data as the vertical axis to construct the y-axis, thereby constructing a load trend chart of the business interface;

[0102] Step S4-3-2: Based on the total number of abnormal calls in each regulatory data collection period T, the x-axis is constructed with the regulatory data collection period T as the horizontal axis, and the total number of abnormal calls as the vertical axis to construct the y-axis, thereby constructing an abnormal trend graph of the business interface.

[0103] In implementation, we traverse the interface list and accumulate the predicted values ​​for each interface. For example, if the system contains 100 business interfaces, we add the predicted call counts for each interface to obtain the total data. We then select an appropriate chart type to construct a trend chart, with the horizontal axis representing the time period and the vertical axis representing the call volume / anomaly volume, to ensure clear trend visualization. Trend chart construction requires cross-period data alignment to avoid trend analysis bias caused by errors in the collection cycle. This can be achieved through millisecond-level synchronization using timestamps.

[0104] Step S5: Setting monitoring thresholds based on the system's load trend and abnormal trend, and using the monitoring thresholds to perform data quality detection on the system;

[0105] Based on the load trend chart and the exception trend chart, the average and variance of the number of service interface calls and the number of call exceptions are calculated. The call count monitoring threshold and the exception count monitoring threshold are set based on the upper limit value according to the Three Sigma principle. The calculation formula for setting the call count monitoring threshold is as follows:

[0106] ;

[0107] Where, E A Indicates the call count monitoring threshold; A avg It is represented by the average number of actual calls to the business interface; n is the number of regulatory data collection cycles T involved in the calculation of the average value; k is the standard deviation multiplier, which is determined by the Three Sigma principle; a i It is expressed as the number of calls during the i-th regulatory data collection cycle T;

[0108] The standard deviation multiplier k is calculated using the following formula:

[0109] ;

[0110] Where, k represents the standard deviation multiple coefficient in the three sigma principle; Amax Indicates the upper limit of the number of business interface calls; A min Indicates the lower limit of the number of business interface calls;

[0111] The calculation formula for setting the abnormality number monitoring threshold is as follows:

[0112] ;

[0113] Where, E B Indicates the monitoring threshold of abnormal number; B avg It is represented by the average number of abnormal business interface calls; b i It is represented by the number of abnormal calls during the i-th regulatory data collection period T;

[0114] The system performs data quality detection based on the call count monitoring threshold and the exception count monitoring threshold. The specific process of data quality detection is as follows:

[0115] When the predicted value of the number of calls exceeds the call number monitoring threshold, a first quality warning signal is issued;

[0116] When the predicted number of calls does not exceed the call number monitoring threshold, the system's business interface calls are continuously monitored;

[0117] When the predicted value of the number of call exceptions exceeds the abnormality monitoring threshold, a second quality warning signal is issued;

[0118] When the predicted value of the number of abnormal calls does not exceed the abnormal number monitoring threshold, the system's business interface calls are continuously monitored.

[0119] In specific implementation, taking the total call count data for 30 consecutive regulatory data collection cycles (T = 1 hour) of a certain regulatory system as an example, the average value μ is calculated to be 8000 times, and the standard deviation σ is 300 times. According to the Three Sigma principle, the upper threshold for call count monitoring is set to μ + 3σ = 8900 times. If the predicted call count for the next cycle is 9200 times, exceeding the threshold, the first quality warning signal is triggered, indicating abnormal system load. If the predicted value is 8500 times, monitoring continues.

[0120] For the number of call anomalies, assuming a historical average of 200 and a standard deviation of 50, the upper threshold for anomaly monitoring is set at 200 + 3 × 50 = 350. If the predicted number of anomalies in a certain period reaches 380, exceeding the threshold, a second quality warning signal is issued, indicating data quality risks. If it reaches 300, continuous monitoring is maintained.

[0121] Example 2, as Figure 2As shown, the present invention provides a supervision data quality monitoring system based on artificial intelligence. The intelligent management system includes a data acquisition and processing module, a data statistical analysis module, a trend prediction and analysis module, a system data integration module and a quality detection and early warning module;

[0122] The data acquisition and processing module is used to set the supervision data acquisition cycle and obtain and standardize the business interface call data; the data statistical analysis module is used to analyze the business interface load data and obtain the number of calls, abnormalities and peak calls of the business interface; the trend prediction and analysis module is used to predict the trend of the number of calls and abnormalities of the business interface; the system data integration module is used to summarize the predicted data of each business interface and construct a load and abnormality trend chart; the quality detection and early warning module is used to set monitoring thresholds and detect and warn the system data quality based on the thresholds;

[0123] The output end of the data acquisition and processing module is electrically connected to the input end of the data statistical analysis module; the output end of the data statistical analysis module is electrically connected to the input end of the trend prediction and analysis module; the output end of the trend prediction and analysis module is electrically connected to the input end of the system data integration module; the output end of the system data integration module is electrically connected to the input end of the quality detection and early warning module;

[0124] The data collection and processing module includes a collection cycle setting unit and a business interface data processing unit; the collection cycle setting unit is used to set a fixed time interval for the supervision data collection cycle; the business interface data processing unit is used to perform a standardized processing flow for cleaning and formatting the acquired business interface call data;

[0125] The data statistics analysis module includes a call data extraction unit and a peak data acquisition unit; the call data extraction unit is used to group and count the number of calls and exceptions of the service interface according to the timestamp and service interface identifier; the peak data acquisition unit is used to obtain the peak number of calls of the service interface by sorting historical data;

[0126] The trend prediction and analysis module includes a call count prediction unit and an exception count prediction unit; the call count prediction unit is used to predict the call count of the service interface based on peak data and a time series model; the exception count prediction unit is used to estimate the exception count of the service interface based on the historical exception ratio and the call count prediction value;

[0127] The system data integration module includes a forecast data aggregation unit and a trend graph construction unit; the forecast data aggregation unit is used to accumulate the call count and exception count prediction values ​​of each business interface to obtain total data; the trend graph construction unit is used to construct a trend graph with the collection period as the horizontal axis and the total data as the vertical axis;

[0128] The quality detection and early warning module includes a monitoring threshold setting unit and a quality early warning management unit; the monitoring threshold setting unit is used to calculate and set the monitoring threshold according to the statistical characteristics of the trend graph through the three sigma principle; the quality early warning management unit is used to compare the predicted value with the threshold and trigger an early warning signal when the threshold is exceeded.

[0129] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be considered in all respects as illustrative and non-restrictive, and the scope of the present invention is defined by the appended claims, not the foregoing description, and all variations within the meaning and range of equivalents of the claims are intended to be included therein. Any reference sign in a claim should not be construed as limiting the claim to which it relates.

Claims

1. A method for monitoring regulatory data quality based on artificial intelligence, characterized by: The regulatory data quality monitoring approach includes the following steps: Step S1: Set a supervision data collection cycle, obtain call data of all business interfaces according to the supervision data collection cycle, and build a business interface load analysis set; Step S2: Analyze the service interface load analysis set to obtain the call count and call exception count data corresponding to the service interface timestamp, obtain the service interface call data for each regulatory data collection period in history, and analyze it to obtain the peak call count of the service interface; Step S3: Analyze and predict the peak call count of the service interface to obtain the call count trend of the service interface, and analyze the call exception ratio trend based on the call exception data of the service interface corresponding timestamp; construct a call exception change trend chart based on the call exception data of the service interface, and combine the call exception trend analysis and prediction to obtain the call exception number of the service interface; Step S3-1: Select the timestamp corresponding to the peak call count as the starting point, obtain the call count data between the service interface starting point and the current time period that is only less than the peak call count, and record it as the second peak call count; Step S3-2: Based on the peak call count and the second peak call count of the business interface, calculate the maximum call count of the business interface in the next regulatory data collection period T through weighted fusion calculation, and record it as the call count prediction threshold; Step S4: Obtain the total number of system calls and total number of call exceptions based on the call number trend and the abnormality ratio trend of each business interface; obtain the system's historical total number of calls and total number of call exceptions to analyze the load trend and abnormality trend of the business interface; Step S5: Setting monitoring thresholds based on the system's load trend and abnormal trend, and using the monitoring thresholds to perform data quality detection on the system; Based on the load trend chart and the exception trend chart, the average and variance of the number of service interface calls and the number of call exceptions are calculated. The call count monitoring threshold and the exception count monitoring threshold are set according to the upper limit value based on the three sigma principle. The calculation formula for setting the call count monitoring threshold is as follows: ； Where EA is the call count monitoring threshold; Aavg is the average call count; n is the number of regulatory data collection cycles T involved in the average value calculation; k is the standard deviation multiplier, which is determined by the three sigma principle and has a value range of 2≤k≤3; ai is the call count in the i-th regulatory data collection cycle T. The calculation formula for setting the abnormality number monitoring threshold is as follows: ； Where EB represents the call count monitoring threshold; Bavg represents the average number of calls; bi represents the number of abnormal calls during the i-th regulatory data collection period T; The system performs data quality detection based on the call count monitoring threshold and the exception count monitoring threshold. The specific process of data quality detection is as follows: When the predicted value of the number of calls exceeds the call number monitoring threshold, a first quality warning signal is issued; When the predicted number of calls does not exceed the call number monitoring threshold, the system's business interface calls are continuously monitored; When the predicted value of the number of call exceptions exceeds the abnormality monitoring threshold, a second quality warning signal is issued; When the predicted value of the number of abnormal calls does not exceed the abnormal number monitoring threshold, the system's business interface calls are continuously monitored.

2. The method for monitoring supervisory data quality based on artificial intelligence according to claim 1, characterized in that: The specific steps of step S1 are as follows: Step S1-1, setting a supervision data collection period T, wherein the supervision data collection period T is a fixed time interval; Step S1-2: Based on the set regulatory data collection cycle T, obtain call data of all business interfaces according to the regulatory data collection cycle T through application program interface (API) calls or log analysis; Step S1-3: standardize the acquired business interface call data, including data cleaning and data format unification; Step S1-4: construct a business interface load analysis set by using the standardized business interface call data with the timestamp as the primary index and the unique identifier of the business interface as the secondary index.

3. The method for monitoring regulatory data quality based on artificial intelligence according to claim 2, characterized in that: The specific steps of step S2 are as follows: Step S2-1: Based on the service interface load analysis set, group and count the service interface call data by timestamp and service interface unique identifier; Step S2-2: Analyze and extract the number of calls and call exceptions for each timestamp of the business interface; Step S2-3: Obtain the call data of the business interface in each supervision data collection period in the history of the business interface, obtain the maximum number of calls of the business interface through bubble sort, and record it as the peak number of calls.

4. The method for monitoring supervisory data quality based on artificial intelligence according to claim 3, characterized in that: The step S3 further includes: Step S3-3: Obtain the call count data for each regulatory data collection cycle T of the business interface history, select a call count prediction threshold as the upper limit of the predicted value of the business interface call count, and use the exponential smoothing method in the time series prediction model to predict the call count of the business interface in the next regulatory data collection cycle T, which is recorded as the call count prediction value; Step S3-4: Perform primary monitoring of the business interface based on the call count prediction threshold. The primary monitoring and early warning judgment is as follows: Step S3-4-1: When the predicted number of calls exceeds the predicted number of calls threshold, a business interface warning signal is issued; Step S3-4-2: When the predicted number of calls does not exceed the predicted number of calls threshold, continue monitoring the business interface; Step S3-5: Divide the number of call exceptions in each supervision data collection period T of the business interface history by the number of calls in the corresponding supervision data collection period T to obtain a call exception ratio; Step S3-6: Calculate the average abnormal call count ratio based on the abnormal call count ratio of each historical regulatory data collection period T of the business interface; Step S3-7: multiply the predicted number of calls of the service interface by the average ratio of abnormal calls to obtain the abnormal number of calls of the service interface, which is recorded as the predicted number of abnormal calls.

5. The artificial intelligence-based supervisory data quality monitoring method according to claim 4, characterized in that: The specific steps of step S4 are as follows: Step S4-1: Using the unique identifier of the service interface as an index, read the predicted number of calls and the predicted number of abnormal calls of each service interface in the system; Step S4-2: accumulating the predicted call times of each business interface in the system to obtain total call times data; accumulating the predicted call times of each business interface in the system to obtain total call times prediction data; Step S4-3: Sort the total number of calls and the total number of abnormal calls in chronological order according to the supervision data collection period T; Step S4-3-1: Based on the total call count data of each regulatory data collection period T, construct the x-axis with the regulatory data collection period T as the horizontal axis and the total call count data as the vertical axis to construct the y-axis, thereby constructing a load trend chart of the business interface; Step S4-3-2: Based on the total number of abnormal calls in each regulatory data collection period T, the x-axis is constructed with the regulatory data collection period T as the horizontal axis, and the total number of abnormal calls as the vertical axis to construct the y-axis, thereby constructing an abnormal trend graph of the business interface.

6. An artificial intelligence-based supervisory data quality monitoring system, applied to the artificial intelligence-based supervisory data quality monitoring method according to any one of claims 1 to 5, characterized in that: The regulatory data quality monitoring system includes data acquisition and processing modules, data statistical analysis modules, trend forecasting and analysis modules, system data integration modules and quality detection and early warning modules; The data acquisition and processing module is used to set the supervision data acquisition cycle and obtain and standardize the business interface call data; the data statistical analysis module is used to analyze the business interface load data and obtain the number of calls, abnormal number and peak number of calls of the business interface; the trend prediction and analysis module is used to predict the trend of the number of calls and abnormal number of calls of the business interface; The system data integration module is used to aggregate the forecast data of each business interface and construct load and abnormality trend graphs; The quality detection and early warning module is used to set monitoring thresholds and detect and warn system data quality based on the thresholds; The output end of the data acquisition and processing module is electrically connected to the input end of the data statistical analysis module; the output end of the data statistical analysis module is electrically connected to the input end of the trend prediction and analysis module; the output end of the trend prediction and analysis module is electrically connected to the input end of the system data integration module; the output end of the system data integration module is electrically connected to the input end of the quality detection and early warning module.

7. The artificial intelligence-based supervisory data quality monitoring system according to claim 6, characterized in that: The data collection and processing module includes a collection cycle setting unit and a business interface data processing unit; the collection cycle setting unit is used to set a fixed time interval for the supervision data collection cycle; the business interface data processing unit is used to perform a standardized processing flow for cleaning and formatting the acquired business interface call data; The data statistics analysis module includes a call data extraction unit and a peak data acquisition unit; the call data extraction unit is used to group and count the number of calls and exceptions of the service interface according to the timestamp and service interface identifier; the peak data acquisition unit is used to obtain the peak number of calls of the service interface by sorting historical data; The trend prediction and analysis module includes a call count prediction unit and an exception count prediction unit; the call count prediction unit is used to predict the call count of the business interface based on peak data and time series model; the exception count prediction unit is used to estimate the exception count of the business interface based on the historical exception ratio and the call count prediction value.

8. The artificial intelligence-based supervisory data quality monitoring system according to claim 6, characterized in that: The system data integration module includes a forecast data aggregation unit and a trend chart construction unit; the forecast data aggregation unit is used to accumulate the call times and exception times prediction values ​​of each business interface to obtain total data; The trend graph construction unit is used to construct a trend graph with the acquisition period as the horizontal axis and the total data as the vertical axis; The quality detection and early warning module includes a monitoring threshold setting unit and a quality early warning management unit; The monitoring threshold setting unit is used to calculate and set the monitoring threshold according to the statistical characteristics of the trend graph through the three sigma principle; the quality warning management unit is used to compare the predicted value with the threshold and trigger an early warning signal when the threshold is exceeded.

Images