Multi-dimensional abnormal data monitoring system

Abstract

The invention discloses a multi-dimensional abnormal data monitoring system, which comprises a data integration module, an anomaly analysis engine, an authority management module, an early warning and response module and a visual display module, and is characterized in that the data integration module performs standardization processing on original data through a dynamic normalization method; the exception analysis engine is combined with single-dimensional rule detection and multi-dimensional clustering analysis to realize rapid screening and deep detection of exceptional data, the authority management module guarantees data security through role and field authority control, and the early warning and response module performs graded early warning and intelligent response on the exceptional data based on an exception score and a trend change rate. And the visual display module visually displays abnormal data distribution and characteristics in forms of a thermodynamic diagram, a radar map and the like. The monitoring system provided by the invention can be suitable for dynamic and changeable scenes, improves the accuracy, response efficiency and user experience of abnormity monitoring, and is applied to the fields of industrial monitoring, gas data monitoring and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of data monitoring and processing, and in particular to a multi-dimensional abnormal data monitoring system. Background Art

[0002] With the wide application of big data technology, various industries face the need for abnormal data monitoring in data processing and analysis. The existence of abnormal data may reflect sudden problems, potential risks or system anomalies in business. Timely and accurately detecting and processing abnormal data is of great significance for an enterprise's risk control and operation optimization. Specifically in the gas enterprise, based on the reported data of user enterprises and stations, through future demand reporting and supply and demand daily reports, multi-dimensional monitoring of abnormal data can be achieved, which can dynamically monitor gas consumption and is of great significance in aspects such as gas resource sharing and gas storage peak shaving management.

[0003] Currently, abnormal data monitoring technologies mainly adopt methods such as rule detection, time series analysis and clustering analysis. Among them: Rule detection simply screens data through fixed upper and lower limits or thresholds, such as judging abnormal points based on rules of mean and standard deviation. Time series analysis detects abnormal points by analyzing the time trend and fluctuations of data in a dynamic scenario. And clustering analysis divides data into multiple categories and judges whether it is abnormal by using the distance between data points and the category center.

[0004] The above-mentioned data monitoring methods in the prior art meet the abnormal data monitoring requirements in specific scenarios to a certain extent. However, with the complexity of business scenarios and the popularity of multi-dimensional data, the following deficiencies have gradually emerged in practical applications:

[0005] 1. Insufficient support for multi-dimensional data. Existing methods usually only process single-dimensional data and ignore the complex relationships between multi-dimensional features, making it difficult to capture abnormal points with cross-dimensional associations.

[0006] 2. Poor adaptability to dynamic environments. The detection method with fixed thresholds cannot adapt to scenarios where data distribution changes over time, easily leading to false alarms or missed alarms.

[0007] 3. Lack of hierarchical evaluation and intelligent response mechanism. Existing systems mostly stay at the simple identification of abnormal points, fail to perform hierarchical processing in combination with the impact degree of abnormal data, and also fail to provide intelligent response measures for different levels of anomalies.

[0008] 4. Insufficient visualization and permission management. The current abnormal data monitoring systems usually lack intuitive visualization means, making it difficult to quickly help users understand the nature and distribution of anomalies. In addition, the permission management for different user roles is weak, posing potential data security risks. Summary of the Invention

[0009] The object of the present invention is to solve the above technical problems and provide a multi-dimensional abnormal data monitoring system. The multi-dimensional abnormal data monitoring system of the present invention is required to achieve the accuracy and comprehensiveness of abnormal detection in complex data scenarios, improve the sensitivity of abnormal handling and resource utilization efficiency, and meet the actual requirements of data security and multi-user scenarios.

[0010] To achieve the above object, the present invention patent is implemented as follows:

[0011] A multi-dimensional abnormal data monitoring system, the system includes:

[0012] Data integration module: used to obtain the original data set X = {x1, x2,..., x n}, and perform normalization processing to generate the standardized data X' = {x'1, x'2,..., x' n};

[0013] Abnormal analysis engine: used to detect data anomalies based on single-dimensional rules and multi-dimensional clustering algorithms;

[0014] Permission management module: used to control the access permissions of different users to data;

[0015] Early warning and response module: used to trigger early warnings and execute response operations according to the abnormal analysis results;

[0016] Visualization display module: used to graphically display the distribution of abnormal data and analysis results.

[0017] As a preferred technical solution of the present invention, the data integration module performs normalization processing on the original data set X through a dynamic normalization formula, and the normalization formula is:

[0018]

[0019] Where: x t represents the t-th data point in the original data; X = {x1, x2,..., x n} is the original data set; is the data mean; λ ∈ [0, 1] is a dynamic adjustment parameter used to balance the influence of the minimum value and the mean value; ∈ is a small positive number to prevent the denominator from being zero, and the output normalized data x' t will be used for subsequent anomaly detection.

[0020] As a preferred technical solution of the present invention, the abnormal analysis engine performs single-dimensional anomaly detection based on the normalized data x' t , and its determination rule is:

[0021] x' t is abnormal if and only if |x' t - μ'| > k · σ' Where: is the mean of the normalized data; is the standard deviation of the normalized data;

[0022] k is a preset threshold for controlling the sensitivity of anomaly detection.

[0023] As a preferred technical solution of the present invention, the anomaly analysis engine further performs multi-dimensional anomaly detection on the preliminarily screened data based on a multi-dimensional clustering algorithm. The clustering center C of the multi-dimensional data points k has the following update formula:

[0024]

[0025] The anomaly determination rule for the data point x is:

[0026] ||x - C k || > γ·r k

[0027] where: S k is the sample set of the k-th class; is the average intra-class distance; γ is a dynamic adjustment coefficient for controlling the anomaly determination threshold.

[0028] As a preferred technical solution of the present invention, the anomaly analysis engine combines the single-dimensional anomaly detection result S uni and the multi-dimensional anomaly detection result S multi into an anomaly score S. The comprehensive scoring formula is:

[0029] S = α·S uni + β·S multi

[0030] where: represents the comprehensive score of the single-dimensional anomaly; represents the comprehensive score of the multi-dimensional anomaly; α, β ∈ [0, 1] are weight parameters and satisfy α + β = 1.

[0031] As a preferred technical solution of the present invention, the warning and response module triggers warnings of different levels according to the anomaly score S. The calculation formula for the warning level L is:

[0032]

[0033] where: Δx t = x′ t - x′ t-1 is the trend change rate of the normalized data; σ′ is the standard deviation of the normalized data;

[0034] Trigger the following responses according to the result of L: L < T1: Send a normal warning; T1 ≤ L < T2: Send an emergency warning and record the log; L ≥ T2: Trigger the automated response mechanism.

[0035] As a preferred technical solution of the present invention, the automated response mechanism includes:

[0036] Roll back the abnormal data and revoke its impact;

[0037] Notify users with specified permissions for manual review;

[0038] Generate an abnormal analysis report, including the detection process and analysis results.

[0039] As a preferred technical solution of the present invention, the permission management module dynamically allocates the permission set P u based on the user role R f and the field permission A u,f :

[0040] P u,f = R u ∩A f

[0041] where: P u,f is the permission set of user u for field f; R u is the permission set of the user role; A f is the authorization set of the field.

[0042] As a preferred technical solution of the present invention, the normalization and anomaly detection module supports dynamic adjustment of parameters λ, k, γ to meet the requirements of anomaly data analysis in different scenarios.

[0043] As a preferred technical solution of the present invention, the visualization display module generates various analysis charts, including abnormal point distribution charts, trend change charts, and multi-dimensional clustering result charts, to visually present the anomaly detection results and their influence ranges.

[0044] Compared with the prior art, the multi-dimensional abnormal data monitoring system of the present invention has the following beneficial effects:

[0045] 1. Through dynamic normalization processing, multi-dimensional feature analysis, and anomaly detection methods that combine single-dimensional and multi-dimensional, the present invention can not only quickly identify anomalies in single-dimensional data but also capture the associated anomalies between multi-dimensional features, significantly improving the accuracy and comprehensiveness of anomaly detection in complex data scenarios and being applicable to dynamically changing data environments.

[0046] 2. Through the comprehensive scoring mechanism and the calculation of the trend change rate, the present invention classifies abnormal data and triggers corresponding early warning and response measures according to the abnormal level. Minor abnormalities are logged, and major abnormalities can trigger an automated response, improving the sensitivity of abnormal handling and the resource utilization efficiency, and providing users with flexible and efficient abnormal management capabilities.

[0047] 3. The present invention displays the distribution and influence range of abnormal data through intuitive heat maps, radar charts, and trend analysis charts, helping users quickly understand the nature of abnormal data. Through the permission management module, refined data access control for different user roles is achieved, ensuring data security and meeting the actual needs of multi-user scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0049] Figure 1 It is a system block diagram of a multi-dimensional abnormal data monitoring system. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] To make the objectives, technical solutions, and advantages of the present invention clearer, the following will clearly and completely describe the technical solutions in the present invention with reference to the drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0051] The following combines the attached Figure 1 drawings to describe the detailed implementation manners of the present invention in detail.

[0052] Embodiment 1:

[0053] This embodiment is applied to the collection and processing of gas data in a certain province. For the collection and analysis of gas data across the province, the focus is on the monitoring of abnormal data. First, it is necessary to integrate and normalize the collected data;

[0054] 1. Data collection and preprocessing: The data integration module obtains the original data set X = {x1, x2,..., x n} from multiple data sources, where x t represents the t-th data point;

[0055] The data may contain missing values, duplicate values, and noise. The processing methods include: Handling missing values: Using the mean imputation method:

[0056]

[0057] where μ represents the mean of the data and n is the total number of data points; Removing duplicates and noise: Removing duplicate data points and abnormal noise points.

[0058] 2. Data normalization processing: Using the dynamic normalization formula to convert the original data set X into a standardized data set X′={x′1,x′2,…,x′ n}, and the formula is as follows:

[0059]

[0060] where: x t : the t-th data point in the original data; X={x1,x2,…,x n}: the data set; min(X), max(X): the minimum and maximum values in the data set; the mean of the data; λ∈[0,1]: the dynamic adjustment parameter, used to balance the influence of the minimum value and the mean on normalization; ∈: a small positive number to prevent the denominator from being zero, usually taking 10 -6 .

[0061] Example: If a sales data set X={100,200,300,400}, then the normalized result X′ is the standardized data set for use by subsequent modules.

[0062] Example 2: On the basis of data collection and processing, perform single-dimensional anomaly detection;

[0063] 1. Anomaly detection logic: Input the normalized data X′={x′1,x′2,…,x′ n}, where x′ t is the t-th normalized data point;

[0064] Calculate the mean and standard deviation of the data:

[0065]

[0066] where: μ′: the mean of the normalized data; σ′: the standard deviation of the normalized data; n: the total number of data points.

[0067] Use the anomaly detection rule: x′ t is abnormal if and only if |x′ t -μ′|>k·σ′,

[0068] where: k: the preset threshold, used to control the sensitivity of anomaly detection, and the typical value is k = 2.5.

[0069] 2. Example detection: After normalizing a certain sales data, X′ = {0.1, 0.2, 0.3, 3.0, 0.4}; calculating the mean μ′ = 0.8 and the standard deviation σ′ = 1.1; it is detected that x′4 = 3.0 satisfies |3.0 - 0.8| > 2.5·1.1, so x′4 is marked as an outlier.

[0070] Example 3: Multi-dimensional outlier detection (based on clustering algorithm);

[0071] 1. Multi-dimensional data input: Multi-dimensional data X′ = {(x′ 1,1 , x′ 1,2 ), …, (x′ n,1 , x′ n,2 )}, each data point has two-dimensional features, such as price and sales volume.

[0072] 2. Clustering analysis process: The initial clustering center C k is randomly selected; using the update formula:

[0073]

[0074] where: The clustering center of the k-th class; S k : The sample set of the k-th class; |S k |: The number of samples in the k-th class.

[0075] Outlier determination rule:

[0076] ||x - C k || > γ·r k

[0077] where: ||x - C k ||: The Euclidean distance between the data point x and the clustering center C k ; The average intra-class distance; γ: The dynamic adjustment coefficient, used to control the outlier determination threshold.

[0078] Example 4: Outlier scoring and warning trigger;

[0079] 1. Outlier scoring calculation: One-dimensional outlier score:

[0080] Multi-dimensional outlier score:

[0081] Comprehensive scoring formula: S = α·S uni + β·S multi ,

[0082] where: S uni : One-dimensional outlier score; Smulti : Multi-dimensional anomaly score; α, β ∈ [0, 1]: Weight coefficients, satisfying α + β = 1.

[0083] 2. Early warning trigger rule: Based on the comprehensive score S and the trend change rate Δx t = x′ t - x′ t-1 , calculate the early warning level:

[0084]

[0085] The range of L determines the early warning level: L < T1: General early warning; T1 ≤ L < T2: Emergency early warning; L ≥ T2: Automatically trigger the response mechanism.

[0086] Example 5: Permission management and visual display;

[0087] 1. Permission management rule Permission allocation rule:

[0088] P u,f = R u ∩ A f

[0089] Where: P u,f : The permission set of user u for field f; R u : The permission set of user roles; A f : The authorization set of fields.

[0090] 2. Visual display:

[0091] The heat map shows the distribution of abnormal data;

[0092] The box plot highlights single-dimensional outliers;

[0093] The radar chart shows the contribution of the feature dimensions of the multi-dimensional anomaly score S multi of.

[0094] The present invention monitors multi-dimensional abnormal data. According to the data filled in by users, the system automatically matches the gas transmission data of the upstream and downstream of the enterprise. Comparing with the data filled in by the enterprise itself, timely early warning is given for the data that cannot be matched, which is convenient for taking manual intervention and correction measures. Enabling the operation commanders to obtain the daily gas usage data of the whole province more sensitively. The data collection and analysis dimensions are more detailed than before. The analysis is more diverse, making the gas usage data of the whole province safer and more accurate. The collection process strictly conducts permission control to avoid data leakage. Avoiding the situation where it was difficult to verify the random filling and reporting by users in the past through the verification mechanism.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-dimensional abnormal data monitoring system, characterized in that: The system includes: Data integration module: used to obtain the original data set X = {x1, x2, ..., x n }, and normalize to generate standardized data X′={x′1,x′2,…,x′ n }; Anomaly analysis engine: used to detect data anomalies based on single - dimension rules and multi - dimension clustering algorithms; Permission management module: used to control the access permissions of different users to data; Early warning and response module: used to trigger early warnings and execute response operations according to the anomaly analysis results; Visualization display module: used to graphically display the distribution of anomaly data and analysis results.

2. A multi-dimensional abnormal data monitoring system as claimed in claim 1, characterized in that: The data integration module normalizes the original data set X through a dynamic normalization formula, and the normalization formula is: Where: x t represents the tth data point in the original data; X = {x1, x2, ..., x n } is the original data set; is the data mean; λ∈[0,1] is a dynamic adjustment parameter used to balance the influence of the minimum value and the mean; ∈ is a small positive number to prevent the denominator from being zero, and the output normalized data x′ t It will be used for subsequent anomaly detection.

3. A multi-dimensional abnormal data monitoring system as claimed in claim 1, characterized in that: The anomaly analysis engine is based on the normalized data x′ t For single-dimensional anomaly detection, the judgment rules are as follows: x′ t Exception, if and only if |x′ t -μ′|>k·σ′where: is the mean of the normalized data; is the standard deviation of the normalized data; k is a preset threshold used to control the sensitivity of anomaly detection.

4. A multi-dimensional abnormal data monitoring system as claimed in claim 3, characterized in that: The anomaly analysis engine further performs multi-dimensional anomaly detection on the initially screened data based on a multi-dimensional clustering algorithm, and the cluster center C of the multi-dimensional data point k The update formula is: The anomaly determination rule for data point x is: ∥xC k ∥>γ·r k Where: S k is the sample set of the kth class; is the average distance within the class; γ is the dynamic adjustment coefficient, which is used to control the abnormality judgment threshold.

5. A multi-dimensional abnormal data monitoring system as claimed in claim 4, characterized in that: The anomaly analysis engine converts the single-dimensional anomaly detection result S uni And the multi-dimensional anomaly detection results S multi The comprehensive score is S, and the comprehensive score formula is: S=α·S uni +β·S multi in: A composite score indicating abnormality in a single dimension; Represents the comprehensive score of multi-dimensional anomalies; α, β∈[0,1] are weight parameters, and α+β=1.

6. A multi-dimensional abnormal data monitoring system as claimed in claim 1, characterized in that: The early warning and response module triggers early warnings of different levels according to the anomaly score S, and the calculation formula for the early warning level L is: Where: Δx t = x′ t -x′ t-1 is the trend change rate of normalized data; σ′ is the standard deviation of normalized data; Trigger the following responses according to the result of L: L < T1: send a general early warning; T1 ≤ L < T2: send an emergency early warning and record the log; L ≥ T2: trigger an automated response mechanism.

7. A multi-dimensional abnormal data monitoring system as claimed in claim 6, characterized in that: The automated response mechanism includes: Roll back the anomaly data and revoke its impact; Notify users with specified permissions for manual review; Generate an anomaly analysis report, including the detection process and analysis results.

8. A multi-dimensional abnormal data monitoring system as claimed in claim 1, characterized in that: The rights management module is based on user role R u and field permissions A f Dynamically assign permission set P u,f : P u,f =R u ∩A f Where: P u,f is the permission set of user u for field f; R u A is the permission set of the user role; f The authorization collection for the field.

9. A multi-dimensional abnormal data monitoring system according to any one of claims 2 to 5, characterized in that: The normalization and anomaly detection module supports dynamic adjustment of parameters λ, k, γ to adapt to the anomaly data analysis requirements of different scenarios.

10. A multi-dimensional abnormal data monitoring system according to claim 1, characterized in that: The visualization display module generates various analysis charts, including anomaly point distribution charts, trend change charts, and multi - dimension clustering result charts, to intuitively present the anomaly detection results and their influence scope.

Images