Automatic monitoring data intelligent auditing method

Abstract

The invention relates to the technical field of data analysis, in particular to an automatic monitoring data intelligent auditing method, which comprises the following steps of: acquiring multi-source data of a plurality of data sources in real time or discontinuously; preprocessing the multi-source data, and performing data self-inspection; performing feature extraction on the to-be-audited data and the historical data, and formulating a corresponding auditing rule; according to the method, large-scale data can be rapidly and accurately processed through the intelligent auditing technology, real-time auditing is achieved, the auditing efficiency and accuracy are improved, the labor and time cost is reduced, the overall economic benefits are improved, the self-optimization function can be achieved, and the method is suitable for popularization and application. According to the new data and the feedback condition of the user, the auditing strategy is continuously optimized to adapt to the change of the environment, and the method further provides flexible rule setting to meet personalized auditing requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of data analysis, and in particular to an automatic monitoring data intelligent audit method. Background Art

[0002] With the rapid development of information technology, manual monitoring data is gradually replaced by automatic detection data, which is crucial for enterprise operation management and decision-making. In order to ensure the quality of data, data auditing has become increasingly important. However, existing data auditing technologies face many challenges.

[0003] Existing audit methods are highly rule-dependent and require manual rule setting, which is time-consuming and cannot adapt to all scenarios. Existing methods also have poor adaptability and low intelligence, making it difficult to adapt to rapidly changing data environments and unable to continuously improve with time and data changes. In solving these problems, high-quality training data is particularly important for building accurate, reliable, and intelligent audit methods, but in actual applications, high-quality data is difficult to obtain. In addition, for some application scenarios, data audits must be conducted in an extreme amount of time, requiring real-time audits.

[0004] In summary, the existing technology has obvious limitations in the field of data audit, especially it is difficult to meet the needs of the growing amount of data and diversified data types. Therefore, it is particularly urgent to develop a new generation of automatic monitoring data intelligent audit technology method that can overcome the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a method for automatically monitoring data intelligent auditing, aiming to solve the problem that the existing technology in the field of data auditing is difficult to meet the needs of the increasing amount of data and diversified data types.

[0006] To achieve the above object, the present invention provides a method for automatically monitoring data intelligent review, comprising the following steps:

[0007] Collect multi-source data in real time or intermittently from multiple data sources;

[0008] Preprocessing the multi-source data and performing data self-checking;

[0009] Extract features from the audited data and historical data, and formulate corresponding audit rules;

[0010] Based on the audit rules, the pre-processed multi-source data is intelligently audited.

[0011] The multi-source data includes automatically monitored resource data, system configuration data, model performance data and work log data.

[0012] The specific method of preprocessing the multi-source data and performing data self-checking is as follows:

[0013] Convert the formats and types of the multi-source data in a unified manner to obtain unified data;

[0014] Performing data self-check on the unified data;

[0015] The unified data is filled with the value by modifying the trend of the previous and next data.

[0016] The specific method of extracting features from the data to be audited and historical data and formulating corresponding audit rules is as follows:

[0017] Extract data features from the data to be audited and the historical data at the points, and perform consistency checks on the audited data and historical data;

[0018] Audit rules are set for the data to be audited based on multiple factors such as literature research, actual conditions and historical data.

[0019] The specific method of performing intelligent audit on the pre-processed multi-source data based on the audit rules is as follows:

[0020] Classify the rules based on historical data features and custom rules into structured rules and unstructured rules, and input them;

[0021] Based on the decision tree, the rules of the audited data are identified, and the intelligent audit rules are selected for audit to obtain the audit results;

[0022] Based on the audit results, the proportion of normal data is calculated, a visual output is performed, and the audit results are reviewed.

[0023] The present invention provides an automatic monitoring data intelligent audit method, which collects multi-source data from multiple data sources in real time or intermittently; pre-processes the multi-source data and performs data self-checking; extracts features from the audited data and historical data, and formulates corresponding audit rules; and performs intelligent audit on the pre-processed multi-source data based on the audit rules. This method uses natural language processing technology and combines human intervention to improve the accuracy and efficiency of data auditing, reduce the influence of certain subjective and objective factors, and enhance the auditing ability of multi-source data. By introducing decision-making mechanisms and self-optimization mechanisms, the audit strategy can be continuously adjusted and optimized in the actual application process to adapt to the ever-changing data environment. This method can reduce the problem of excessive labor and time costs caused by large-scale data and complex data types, and can cope with more complex application scenarios. It has good adaptability and security. Through intelligent audit technology, it can realize fast and accurate processing of large-scale data, realize real-time audit, improve audit efficiency and accuracy, reduce labor and time costs, and improve overall economic benefits. It can realize self-optimization function and continuously optimize audit strategies to adapt to environmental changes based on new data and user feedback. This method also provides flexible rule settings to meet personalized audit needs, solving the problem that existing technologies in the field of data audit are difficult to meet the needs of growing data volumes and diversified data types. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 This is a schematic diagram of the 3 sigma principle (Laida criterion).

[0026] Figure 2 It is a schematic diagram of the intelligent audit mechanism.

[0027] Figure 3 The present invention provides a flowchart of an automatic monitoring data intelligent review process.

[0028] Figure 4 The present invention provides a flow chart of an automatic monitoring data intelligent audit method.

[0029] Figure 5 It is a flowchart of the specific method of extracting features from audit data and historical data and formulating corresponding audit rules.

[0030] Figure 6It is a flowchart of a specific method for performing data preprocessing on the multi-source data.

[0031] Figure 7 It is a flowchart of a specific method of performing intelligent audit on pre-processed multi-source data based on the audit rules. DETAILED DESCRIPTION

[0032] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0033] See also Figures 1 to 7 ,The present invention provides an automatic monitoring data intelligent audit method, comprising the following steps;

[0034] S1 collects multi-source data from multiple data sources in real time or intermittently;

[0035] In the embodiment of the present invention, multi-source data is collected from multiple data sources in real time or intermittently, and the multi-source data includes automatically monitored resource data, system configuration data, model performance data, work log data, etc. Different data collection intervals and methods are set according to different data characteristics to collect data to ensure the accuracy and real-time performance of data collection, and the collected data is transmitted to the storage through various methods such as database, API interface and data file transmission.

[0036] S2 preprocesses the multi-source data and performs data self-check;

[0037] Specific method:

[0038] S21 uniformly converts the formats and types of the multi-source data to obtain unified data;

[0039] In the embodiment of the present invention, since the formats and types of multi-source data collected through data collection are different, it is necessary to convert the original data into a unified format (such as: .csv, etc.) and data type (such as: date, string, etc.), that is, to unify the data to facilitate subsequent experiments.

[0040] S22 performs data self-check on the unified data;

[0041] In an embodiment of the present invention, the Pandas library in Python is used to monitor duplicate values, missing values, and abnormal values, and the proportion of the overall data is calculated as a preliminary audit result of the group of data.

[0042] The 3sigma principle is used to screen outliers, that is, data values ​​that deviate from the overall mean by more than 3 standard deviations become highly abnormal outliers. The specific principles are: ① 68% of the data fall within plus or minus one standard deviation of the mean; ② 95% of the data fall within plus or minus two standard deviations of the mean; ③ 99.7% of the data fall within plus or minus three standard deviations of the mean. ④ Data that fall outside plus or minus three standard deviations of the mean are outliers.

[0043] S23 fills the value of the unified data by modifying the trend of the previous and next data.

[0044] In an embodiment of the present invention, in order to maintain the integrity and accuracy of the time series, it is necessary to process the detected duplicate values, missing values ​​and abnormal values, and choose to use the average value or median of the audited data to fill them, or use a more accurate method, that is, fill the value by modifying the trend of the previous and next data of the value.

[0045] S3 extracts features from the data to be audited and historical data, and formulates corresponding audit rules;

[0046] Specific method:

[0047] S31 extracts data features from the data to be audited and the historical data at the point, and performs consistency check on the audited data and the historical data;

[0048] In the embodiment of the present invention, historical data with good quality at the audit data point is used to extract data features, calculate the mean, standard deviation, median and other data features of the historical data, and perform rule audit.

[0049] The ICC intraclass correlation coefficient is a method for data consistency testing. This method is suitable for quantitative and categorical data. By performing a consistency test on the audit data and historical data, it can be determined whether the theoretical distribution and rules of the two sets of data are consistent. The value of this coefficient is in the range of [0, 1], representing different degrees of consistency.

[0050] Table 1 ICC consistency level classification

[0051] ICC coefficient Degree of consistency ICC<0.2 Poor 0.2<ICC<0.4 generally 0.4<ICC<0.6 medium 0.6<ICC<0.8 Strong 0.8<ICC<1 powerful

[0052] The calculation formula of ICC intraclass correlation coefficient is as follows:

[0053] ICC=(MSB-MSW) / (MSB+(k-1))×MSW)

[0054] Among them, MSB represents the mean square error between groups, MSW represents the mean square error within groups, and k represents the number of samples.

[0055] S32 sets audit rules for the data to be audited based on multiple factors such as literature research, actual conditions and historical data.

[0056] In the embodiment of the present invention, through literature research, historical experience and actual conditions, appropriate audit rules are manually set for all data to be audited, and the audit is performed.

[0057] S4 performs intelligent audit on the pre-processed multi-source data based on the audit rules.

[0058] Specific method:

[0059] S41 classifies the rules based on historical data features and the user-defined rules into structured rules and unstructured rules, and inputs them;

[0060] In the embodiment of the present invention, the rules based on historical data and custom rules are classified into structured rules and unstructured rules, and these data are input. Then, the audit data is statistically analyzed through the Pandas library in Python, and corresponding data extraction is performed to facilitate data audit.

[0061] S42 identifies rules for the audited data based on the decision tree, selects intelligent audit rules for audit, and obtains audit results;

[0062] In the embodiment of the present invention, a decision tree is used to identify rules for data to be audited. First, the rules for intelligent auditing are selected through the audit mechanism, and then normal data that meets the conditions is screened through the corresponding rules.

[0063] S43 calculates the proportion of normal data based on the audit results, performs visual output, and reviews the audit results.

[0064] In the embodiment of the present invention, the proportion of normal data is calculated through intelligent audit results, and visual output is performed to review the audit results. If the audit regulations are not met, new rules are set through feedback information for re-audit. The audited data is output to the database for later use; and the data feature information of the audited data is input into the rule library for use in the next audit.

[0065] What is disclosed above is only a preferred embodiment of the method for automatic monitoring data intelligent auditing of the present invention. Of course, it cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiments and equivalent changes made according to the claims of the present invention are still within the scope of the invention.

Claims

1. A method for intelligent auditing of automatic monitoring data, characterized in that: The steps include: Collect multi-source data in real time or intermittently from multiple data sources; Preprocessing the multi-source data and performing data self-checking; Extract features from the audited data and historical data, and formulate corresponding audit rules; Based on the audit rules, the pre-processed multi-source data is intelligently audited.

2. The method for intelligent auditing of automatic monitoring data according to claim 1, characterized in that; The multi-source data includes automatically monitored resource data, system configuration data, model performance data and work log data.

3. The automatic monitoring data intelligent audit method according to claim 1, characterized in that; The specific method of preprocessing the multi-source data and performing data self-checking is: Convert the formats and types of the multi-source data in a unified manner to obtain unified data; Performing data self-check on the unified data; The unified data is filled with the value by modifying the trend of the previous and next data.

4. The automatic monitoring data intelligent audit method according to claim 1, It is characterized by: The specific method of extracting features from the audited data and historical data and formulating corresponding audit rules is as follows: Extract data features from the audited data and historical data at the points, and perform consistency checks on the audited data and historical data; Audit rules are set for the data to be audited based on multiple factors such as literature research, actual conditions and historical data.

5. The automatic monitoring data intelligent audit method according to claim 1, It is characterized by: The specific method of performing intelligent audit on the pre-processed multi-source data based on the audit rules is as follows: Classify the rules based on historical data features and customized rules into structured rules and unstructured rules, and input them; Based on the decision tree, the rules of the audited data are identified, and the intelligent audit rules are selected for audit to obtain the audit results; Based on the audit results, the proportion of normal data is calculated, a visual output is performed, and the audit results are reviewed.

Images