Big data platform management system and method

Through the preprocessing, data calculation and monitoring and early warning module of the big data platform management system, the data quality dependence and false alarm problems in abnormal detection are solved, and accurate abnormal detection and real-time early warning are realized in complex environments, reducing operations risk.

CN119027051AInactive Publication Date: 2024-11-26JIANGXI YIFUOU TECH CO LTD

Patent Information

Application Number
CN202411098430.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing big data platform management system has data quality dependence, false alarms and missed reports in abnormal detection, making it difficult to detect abnormal situations under specific conditions, and lacks real-time monitoring and early warning mechanisms.

Method used

A big data platform management system is designed, including a preprocessing module, a data calculation module and a monitoring and early warning module. Through data cleaning, feature selection, model establishment and real-time monitoring and early warning, abnormal data are screened, abnormal scores are calculated, and early warnings are issued in a timely manner.

Benefits of technology

It realizes accurate abnormality detection when data distribution is damaged and noise, can effectively separate abnormal data from normal data, reduce business operation risks, and improve the accuracy and real-time nature of abnormality detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027051A_ABST
    Figure CN119027051A_ABST
Patent Text Reader

Abstract

The invention is applicable to the field of big data management, and provides a big data platform management system which comprises a preprocessing module, a data calculation module, a model establishment module and a monitoring and early warning module. Through the algorithm provided by the invention, the complex anomaly detection problem can be better processed. Even if data distribution is damaged and large noise exists, a relatively accurate and stable anomaly detection model can be given, anomaly detection can be effectively carried out, and a set of known normal data is not needed, so that unknown anomaly conditions can be better coped with, new data is classified, and the detection efficiency is improved. And abnormal data and normal data can be effectively separated. In the abnormal data detection preprocessing link, abnormal samples can be accurately screened out and marked. And real-time monitoring and early warning can be carried out, timely alarm can be carried out, and risks and loss of business operation and management caused by data abnormity can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of big data management, and in particular relates to a big data platform management system and method. Background Art

[0002] Big data, or massive data, refers to the amount of data involved that is so large that it cannot be captured, managed, processed, and organized into information that helps companies make more proactive business decisions within a reasonable time through mainstream software tools.

[0003] When using a big data platform management system for data analysis, anomaly detection is one of the important functions. The purpose of anomaly detection is to discover anomalies that are inconsistent with normal data, so that problems that need to be monitored and handled can be more keenly discovered. In practice, there are many ways to implement anomaly detection, such as methods based on statistical models, methods based on machine learning models, and so on. But no matter which method is used, anomaly detection itself has certain defects.

[0004] First, anomaly detection often relies on data quality. If errors occur during data collection and processing, the results of anomaly detection may be inaccurate. Second, in some cases, anomalies only occur under certain conditions, which may not appear in the sample, making it difficult to detect these anomalies.

[0005] In addition, anomaly detection also has the problems of false positives and false negatives. False positives refer to the misclassification of normal situations as abnormal situations, which leads to unnecessary actions. Therefore, it is necessary to find a way to accurately grasp abnormal situations. False negatives refer to the failure to find abnormal situations, which leads to the failure to solve the problem in a timely manner. Summary of the invention

[0006] The purpose of the embodiments of the present invention is to provide a big data platform management system, aiming to solve the technical problems existing in the prior art identified in the background technology.

[0007] The embodiment of the present invention is implemented as follows: a big data platform management system, the system comprising:

[0008] The preprocessing module is used to clean and filter the data and filter out erroneous or invalid data;

[0009] A data calculation module is used to select an algorithm and calculate each potential abnormal data to obtain an abnormal score of the potential abnormal data;

[0010] The model building module is used to build a data analysis model and evaluate the data status in combination with the abnormality score obtained by the data calculation module;

[0011] The monitoring and early warning module is used to monitor the evaluation results of the model building module in real time and establish early warning indicators, compare the evaluation results of the model building module with the early warning indicators, and issue data warnings when data exceeding the indicators appears.

[0012] As a further solution of the present invention, the preprocessing module includes:

[0013] Data cleaning unit, used to remove and repair missing values, duplicate values, abnormal values ​​and erroneous values ​​of the collected data;

[0014] A data conversion unit, used to uniformly convert data in different formats according to the current storage environment;

[0015] Data integration unit, which is used to merge data from different sources into one data set and merge and remove duplicate fields;

[0016] Data reduction to simple elements is used to reduce the amount of data storage and processing time while retaining the main features of the data;

[0017] The feature selection unit is used to select features related to the analysis target from a large amount of data to improve the accuracy of modeling.

[0018] As a further solution of the present invention, the model building module includes:

[0019] A model building unit is used to build a model according to the form of data, and optimize and verify the built model;

[0020] A machine learning unit, for training the established model using artificial intelligence and machine learning algorithms in combination with the anomaly scores obtained in the data calculation module;

[0021] The data analysis unit is used to analyze the data based on the abnormal scores obtained in the comprehensive data calculation module.

[0022] As a further solution of the present invention, the monitoring and early warning module includes:

[0023] The indicator establishment unit is used to establish data early warning indicators in combination with the learning data of the model establishment module;

[0024] The data comparison unit is used to compare the data analysis results obtained by the model building module with the data warning index. When the analysis results exceed the data warning index, the data is marked and a warning notification is issued;

[0025] The module optimization unit is used to organize the monitoring and early warning results and optimize the algorithms and models.

[0026] Another object of an embodiment of the present invention is to provide a big data platform management method, the method comprising:

[0027] Clean and filter the data to filter out erroneous or invalid data;

[0028] Select an algorithm and calculate each potential abnormal data to obtain an abnormal score of the potential abnormal data;

[0029] Establish a data analysis model and evaluate the data status based on the anomaly score obtained by the data calculation module;

[0030] Monitor the evaluation results of the model building module in real time and establish early warning indicators. Compare the evaluation results of the model building module with the early warning indicators, and issue data warnings when data exceeding the indicators appears.

[0031] As a further solution of the present invention, the cleaning and screening of data to filter out erroneous or invalid data includes the following steps:

[0032] Remove and repair missing, duplicate, outlier, and erroneous values ​​from the collected data;

[0033] According to the current storage environment, data in different formats are uniformly converted;

[0034] Combine data from different sources into one data set and merge and remove duplicate fields;

[0035] Reduce the amount of data stored and the processing time while retaining the main features of the data;

[0036] Select features related to the analysis target from a large amount of data to improve the accuracy of modeling.

[0037] As a further solution of the present invention, the establishment of the data analysis model and the evaluation of the data status in combination with the abnormality score obtained by the data calculation module include the following steps:

[0038] Establish models according to the form of data, and optimize and verify the established models;

[0039] Used to train the established model using artificial intelligence and machine learning algorithms combined with the anomaly scores obtained in the data calculation module;

[0040] The data is analyzed using anomaly scores obtained in the comprehensive data calculation module.

[0041] As a further solution of the present invention, the evaluation results of the real-time monitoring model building module and the establishment of early warning indicators, and the comparison of the evaluation results of the model building module with the early warning indicators, include the following steps:

[0042] Combine the learning data of the model building module to establish data warning indicators;

[0043] Compare the data analysis results obtained by the model building module with the data warning indicators. When the analysis results exceed the data warning indicators, mark the data and issue a warning notification;

[0044] Organize the results of monitoring and early warning, and optimize the algorithms and models.

[0045] The beneficial effects of the embodiments of the present invention are:

[0046] The algorithm provided by the present invention can better handle complex anomaly detection problems. Even when the data distribution is damaged and there is a lot of noise, a relatively accurate and stable anomaly detection model can be given, anomaly detection can be effectively performed, and a known set of normal data is not required. Therefore, it can better deal with unknown abnormal situations, classify new data, and effectively separate abnormal data from normal data.

[0047] In the abnormal data detection preprocessing phase, abnormal samples can be accurately screened and marked. In addition, real-time monitoring and early warning can be carried out, and alarms can be issued in time to reduce the risks and losses caused by data anomalies to business operations and management. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A structural block diagram of a big data platform management system provided by an embodiment of the present invention;

[0049] Figure 2 A structural block diagram of a preprocessing module provided in an embodiment of the present invention;

[0050] Figure 3 A structural block diagram of a model building module provided in an embodiment of the present invention;

[0051] Figure 4 A structural block diagram of a monitoring and early warning module provided in an embodiment of the present invention;

[0052] Figure 5 A flowchart of a big data platform management method provided by an embodiment of the present invention;

[0053] Figure 6 A flowchart of cleaning and screening data to filter out erroneous or invalid data provided by an embodiment of the present invention;

[0054] Figure 7 A flowchart of establishing a data analysis model provided by an embodiment of the present invention and evaluating the data status in combination with an abnormality score obtained by a data calculation module;

[0055] Figure 8 The present invention provides a flowchart for monitoring the evaluation results of the model building module in real time and establishing early warning indicators, and comparing the evaluation results of the model building module with the early warning indicators. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0057] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, without departing from the scope of this application, a first xx script may be referred to as a second xx script, and similarly, a second xx script may be referred to as a first xx script.

[0058] Figure 1 The structural block diagram of the big data platform management system provided by the embodiment of the present invention is as follows: Figure 1 As shown, a big data platform management system, characterized in that the system includes:

[0059] The preprocessing module 100 is used to clean and filter the data and filter out erroneous or invalid data;

[0060] In this module, data cleaning may include spell checking, unified coding, etc. Depending on the specific situation, other steps may also be added, such as data sampling, data visualization, etc. Each step requires careful consideration and implementation to ensure the rationality and accuracy of the data, so as to effectively improve the accuracy and reliability of data analysis results.

[0061] The data calculation module 200 is used to select an algorithm and calculate each potential abnormal data to obtain an abnormality score of the potential abnormal data;

[0062] In this module, when selecting a data anomaly detection algorithm, the goal is to find an algorithm that can effectively identify abnormal data to help better understand the data and make correct decisions. Different algorithms are suitable for different types of data and different abnormal situations. Therefore, it is necessary to select an algorithm based on the characteristics of the data set and the abnormal situation.

[0063] In terms of calculation, the algorithm needs to calculate different indicators. In the data preprocessing stage, data cleaning, data normalization, data discretization and other operations may be required to better apply the algorithm. In the modeling algorithm, parameter adjustment and model optimization need to be performed according to the specific model.

[0064] The model building module 300 is used to build a data analysis model and evaluate the data status in combination with the abnormality score obtained by the data calculation module;

[0065] In this module, once the model is built, it needs to be optimized. This process includes parameter adjustment, variable selection, feature engineering, etc.

[0066] It is very important to validate the model, which can be done through methods such as cross-validation and leave-one-out validation. Validating the model can ensure that the model has a certain degree of accuracy and generalization.

[0067] Finally, the optimized model is applied to practical problems. In practical applications, it is necessary to consider how to embed the model into the business system, how to monitor and evaluate the model, and other issues.

[0068] The monitoring and early warning module 400 is used to monitor the evaluation results of the model building module in real time and establish early warning indicators, compare the evaluation results of the model building module with the early warning indicators, and issue data early warnings when data exceeding the indicators appears.

[0069] In this module, the purpose of real-time monitoring and early warning is to promptly discover and resolve potential abnormal problems. Through real-time monitoring, it is possible to ensure that errors or abnormalities are known early before they occur, and intervene quickly to reduce risks and losses. In addition, real-time monitoring can also help enterprises identify system problems, such as performance bottlenecks, configuration errors, hardware failures, etc., to ensure the normal operation of the system and improve efficiency.

[0070] Figure 2 The structural block diagram of the preprocessing module provided in the embodiment of the present invention is as follows: Figure 2 As shown, the preprocessing module 100 includes:

[0071] A data cleaning unit 110 is used to delete and repair missing values, duplicate values, abnormal values ​​and erroneous values ​​of the collected data;

[0072] In this unit, data cleaning refers to removing dirty data, duplicate data, invalid data and other data that do not meet the requirements in the data set, so as to reduce and avoid interference with the accuracy of model training and subsequent analysis results. Methods for cleaning data generally include data deduplication, data filling, data filtering and other operations. When cleaning data, factors such as the source of the data, data quality and data processing process need to be considered.

[0073] The data conversion unit 120 is used to uniformly convert data in different formats according to the current storage environment;

[0074] In this unit, data conversion is the process of converting data from one format to another to meet the needs of different analysis requirements. Data conversion methods include data reconstruction, data normalization, data discretization, data encoding and other operations. When performing data conversion, factors such as data type, data quality, and the purpose of data processing need to be considered.

[0075] A data integration unit 130, for merging data from different sources into one data set, and merging and removing duplicate fields;

[0076] In this unit, data integration refers to the integration of data from different sources, formats, and contents to achieve more comprehensive and accurate data analysis. When performing data integration, key issues need to be considered, such as data format conversion, data relationship processing, and data redundancy removal.

[0077] Data is simplified to 140, which is used to reduce the storage volume and processing time of data while retaining the main features of the data;

[0078] In this unit, data reduction refers to compressing data into smaller data sets for faster and more efficient analysis. Data reduction methods include attribute subset selection, dimensionality reduction, and other operations. When performing data reduction, factors such as data stability, data information loss, and the purpose of data processing need to be considered.

[0079] The feature selection unit 150 is used to select features related to the analysis target from a large amount of data to improve the accuracy of modeling.

[0080] In this unit, data feature selection refers to merging and summarizing data to obtain more comprehensive and accurate analysis results. Data feature selection methods include data aggregation, data merging, data transformation and other operations. When performing data feature selection, factors such as data accuracy, data fluctuation, and data relationships need to be considered.

[0081] By calculating the data anomaly detection problem through this algorithm, a set of anomaly scores will be obtained, which can be used to measure the relative abnormality between each data point and normal data. Generally speaking, the data point with higher scores is more likely to be an outlier. These scores can then be compared with a set threshold to determine whether the data point is considered an outlier.

[0082] The algorithm determines whether the data is abnormal by calculating the distance from the data point to the hyperplane. The data points are mapped to a high-dimensional space, and a hyperplane is constructed in this high-dimensional space to separate normal data points from abnormal data points. The position of the hyperplane is determined by maximizing the optimization interval, and then the abnormal data points are identified and determined.

[0083] In this algorithm, commonly used kernel functions include linear kernel, polynomial kernel and Gaussian kernel. The choice of kernel function has a great influence on accuracy and speed. In practical applications, it is necessary to select the most appropriate kernel function and other parameters according to the specific problem.

[0084] Figure 3 The structural block diagram of the model building module provided by the embodiment of the present invention is as follows: Figure 3 As shown, the model building module 300 includes:

[0085] A model building unit 310 is used to build a model according to the form of data, and optimize and verify the built model;

[0086] In this unit, the above algorithm does not directly determine which data points are outliers, but instead provides an anomaly score for each data point. It is necessary to interpret these scores through the established model, combined with application scenarios and domain knowledge, and determine which scores are above the threshold.

[0087] A machine learning unit 320, for training the established model using artificial intelligence and machine learning algorithms in combination with the anomaly scores obtained in the data calculation module;

[0088] In this unit, the main purpose of artificial intelligence and machine learning in data anomaly detection is to allow the model to automatically discover and identify abnormal values ​​or abnormal data points, and adjust and optimize the model based on these abnormal information to improve its accuracy and stability, so that it can better identify future abnormal scores. Therefore, machine learning is an iterative process, and it continuously updates its internal model to adapt to new data, thereby achieving more accurate data anomaly detection.

[0089] The data analysis unit 330 is used to analyze the data based on the abnormality scores obtained in the comprehensive data calculation module.

[0090] In this unit, the established model can be used to evaluate and judge the anomaly scores of all data points, so as to determine whether each data point is an anomaly. Specifically, the threshold can be set to determine which scores are abnormal and which scores are normal. For example, when the anomaly score is greater than a certain set threshold, the data point is classified as an anomaly. Therefore, this algorithm can be used in anomaly detection problems to identify potential abnormal data points by judging and classifying the anomaly scores of data points.

[0091] Figure 4 The structural block diagram of the monitoring and early warning module provided by the embodiment of the present invention is as follows: Figure 4 As shown, the monitoring and early warning module 400 includes:

[0092] An indicator establishment unit 410, used to establish a data early warning indicator in combination with the learning data of the model establishment module;

[0093] The data comparison unit 420 is used to compare the data analysis results obtained by the model building module with the data warning index. When the analysis results exceed the data warning index, the data is marked and a warning notification is issued;

[0094] In this unit, according to the model analysis results, the corresponding threshold is set, and the warning mechanism is triggered when the abnormal data exceeds the set threshold. The warning information can be realized in a variety of ways, such as SMS, email, instant messaging, etc.

[0095] The module optimization unit 430 is used to organize the monitoring and early warning results and optimize the algorithms and models.

[0096] In this unit, the thresholds and models of monitoring and early warning need to be regularly updated and adjusted to ensure the effectiveness and reliability of the monitoring mechanism. At the same time, the results of monitoring and early warning should be fed back and sorted out in a timely manner, and the algorithms and models should be optimized to prepare for the next round of monitoring and early warning.

[0097] Figure 5 A flowchart of a big data platform management method provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, a big data platform management method is characterized in that the method includes:

[0098] S100, cleaning and screening the data to filter out erroneous or invalid data;

[0099] In this step, data cleaning may include spelling check, unified coding, etc. Depending on the specific situation, other steps may be added, such as data sampling, data visualization, etc. Each step requires careful consideration and implementation to ensure the rationality and accuracy of the data, so as to effectively improve the accuracy and reliability of data analysis results.

[0100] S200, selecting an algorithm and calculating each potential abnormal data to obtain an abnormality score of the potential abnormal data;

[0101] In this step, when selecting a data anomaly detection algorithm, the goal is to find an algorithm that can effectively identify abnormal data to help better understand the data and make correct decisions. Different algorithms are suitable for different types of data and different abnormal situations. Therefore, it is necessary to select an algorithm based on the characteristics of the data set and the abnormal situation.

[0102] In terms of calculation, the algorithm needs to calculate different indicators. In the data preprocessing stage, data cleaning, data normalization, data discretization and other operations may be required to better apply the algorithm. In the modeling algorithm, parameter adjustment and model optimization need to be performed according to the specific model.

[0103] S300, establishing a data analysis model, and evaluating the data status in combination with the abnormality score obtained by the data calculation module;

[0104] In this step, once the model is built, it needs to be optimized. This process includes parameter adjustment, variable selection, feature engineering, etc.

[0105] It is very important to validate the model, which can be done through methods such as cross-validation and leave-one-out validation. Validating the model can ensure that the model has a certain degree of accuracy and generalization.

[0106] Finally, the optimized model is applied to practical problems. In practical applications, it is necessary to consider how to embed the model into the business system, how to monitor and evaluate the model, and other issues.

[0107] S400, real-time monitoring of the evaluation results of the model building module and establishing early warning indicators, comparing the evaluation results of the model building module with the early warning indicators, and issuing data warnings when data exceeding the indicators appear.

[0108] In this step, the purpose of real-time monitoring and early warning is to promptly discover and resolve potential abnormal problems. Through real-time monitoring, it is possible to ensure that errors or abnormalities are known early before they occur, and intervene quickly to reduce risks and losses. In addition, real-time monitoring can also help enterprises identify system problems, such as performance bottlenecks, configuration errors, hardware failures, etc., to ensure the normal operation of the system and improve efficiency.

[0109] Figure 6 A flowchart of cleaning and filtering data to filter out erroneous or invalid data is provided in an embodiment of the present invention, such as Figure 6 As shown, the cleaning and screening of data to filter out erroneous or invalid data includes the following steps:

[0110] S110, delete and repair missing values, duplicate values, outliers and erroneous values ​​of the collected data;

[0111] In this step, data cleaning refers to removing dirty data, duplicate data, invalid data and other data that do not meet the requirements in the data set, so as to reduce and avoid interference with the accuracy of model training and subsequent analysis results. The methods of data cleaning generally include data deduplication, data filling, data filtering and other operations. When performing data cleaning, factors such as the source of the data, data quality and data processing process need to be considered.

[0112] S120, converting data in different formats uniformly according to the current storage environment;

[0113] In this step, data conversion is to convert data from one format to another to meet the needs of different analysis requirements. Data conversion methods include data reconstruction, data normalization, data discretization, data encoding and other operations. When performing data conversion, factors such as data type, data quality, and the purpose of data processing need to be considered.

[0114] S130, merging data from different sources into one data set, and merging and removing duplicate fields;

[0115] In this step, data integration refers to the integration of data from different sources, formats, and contents to achieve more comprehensive and accurate data analysis. When performing data integration, key issues need to be considered, such as data format conversion, data relationship processing, and data redundancy removal.

[0116] S140, reducing the storage volume and processing time of data while retaining the main features of the data;

[0117] In this step, data reduction refers to compressing data into smaller data sets for faster and more efficient analysis. Data reduction methods include attribute subset selection, dimension reduction, and other operations. When performing data reduction, factors such as data stability, data information loss, and the purpose of data processing need to be considered.

[0118] S150, selects features related to the analysis target from a large amount of data to improve the accuracy of modeling.

[0119] In step S150, features related to the analysis target are selected from a large amount of data to improve the accuracy of modeling, which is achieved through a feature selection function;

[0120] The feature selection function corresponds to the following function expression:

[0121]

[0122] Among them, F(X) represents the output of the feature selection function, which is used to quantify the relevance of the entire feature set to the analysis target; X represents the set of input data, that is, the feature vector; n represents the number of features in the feature set, i represents the feature sequence number, ω i Represents feature x i The weight, T i Represents feature x i Correlation coefficient with the target, μ i Represents feature x iThe average value of : σ i Represents feature x i The standard deviation is used to measure the dispersion of the data.

[0123] As a supplementary note, The part is used to measure the difference between the feature value and its mean. The greater the difference, the greater the value of this item, indicating that this feature may contain more information. The part is a Gaussian function, which is used to smooth the difference between the eigenvalue and its mean to prevent extreme values ​​from having too great an impact on the result.

[0124] In this step, data feature selection refers to merging and summarizing data to obtain more comprehensive and accurate analysis results. Data feature selection methods include data aggregation, data merging, data transformation and other operations. When performing data feature selection, factors such as data accuracy, data fluctuation, and data relationship need to be considered.

[0125] In step S200, an algorithm is selected and each potential abnormal data is calculated to obtain an abnormality score of the potential abnormal data, which includes the following steps:

[0126] Step S201, confirm each potential abnormal data and obtain the features contained in each potential abnormal data:

[0127] Step S202, obtaining a feature value corresponding to each feature among all features contained in each potential abnormal data;

[0128] Step S203, calculating the anomaly score of the potential abnormal data according to the feature value corresponding to each feature of all features included in the potential abnormal data, the average value of the feature and the standard deviation of the feature.

[0129] Among them, the function expression corresponding to the anomaly score of potential abnormal data is:

[0130]

[0131] d=(f1,f2,...,f i …, f m )

[0132] Among them, A(d) represents the anomaly score of potential anomaly data d, f i represents the feature value of the i-th feature of the potential abnormal data d, and m represents the number of features contained in the potential abnormal data d.

[0133] It should be noted here that In the , we calculate the square root of the sum of the fourth power of the standardized deviations of each feature value from its mean. This section highlights the impact of feature values ​​far from the mean on the anomaly score, because anomalies are usually far from the normal range. In the first part, the product of the relative deviations of all eigenvalues ​​from their mean is calculated, further enhancing the influence of those eigenvalues ​​that are significantly different from the mean on the anomaly score.

[0134] As a supplementary note, the purpose of the design of the anomaly score function for potential anomaly data is to comprehensively consider the various features of the data and calculate an overall anomaly score by highlighting the abnormal features. The higher the score, the more likely the data point is to be an anomaly.

[0135] Figure 7 The flowchart of establishing a data analysis model provided by an embodiment of the present invention and evaluating the data status in combination with the abnormality score obtained by the data calculation module is as follows: Figure 7 As shown, the establishment of the data analysis model and the evaluation of the data status in combination with the abnormality score obtained by the data calculation module include the following steps:

[0136] S310, establishing a model according to the form of the data, and optimizing and verifying the established model;

[0137] In this step, the algorithm does not directly determine which data points are outliers, but provides an anomaly score for each data point. The established model needs to be combined with application scenarios and domain knowledge to interpret these scores and determine which scores are above the threshold.

[0138] S320, using artificial intelligence and machine learning algorithms, and training the established model in combination with the anomaly scores obtained in the data calculation module;

[0139] In step S320, in order to adopt artificial intelligence and machine learning algorithms, and to train the established model in combination with the anomaly score obtained in the data calculation module, a complex loss function is used, which takes the anomaly score as input and is used to optimize the model parameters.

[0140] Among them, the functional expression of the loss function is:

[0141]

[0142] Among them, L(θ, X, y) represents the loss function, f θ (x k ) represents the model's response to the input data x k The predicted value, y represents the actual target value, x k represents the kth input data point, y k Represents x kThe corresponding true target value, N represents the number of samples in the data set, A(x k ) represents the input data point x k anomaly score, θ represents the parameter set of the model; α represents the first positive hyperparameter, which is used to control the weight of the anomaly score item in the loss function; β represents the second positive hyperparameter, which is used to control the nonlinear effect of the anomaly score on the loss function.

[0143] In the present invention, the purpose of the above loss function is to introduce a weighted term related to the anomaly score on the basis of the standard mean square error loss, so that the model pays more attention to those data points with higher anomaly scores. In this way, the model can automatically identify and adapt to these anomalies during the training process, thereby improving the accuracy of predicting future anomaly scores.

[0144] In this step, the main purpose of artificial intelligence and machine learning in data anomaly detection is to allow the model to automatically discover and identify abnormal values ​​or abnormal data points, and adjust and optimize the model based on these abnormal information to improve its accuracy and stability, so that it can better identify future abnormal scores. Therefore, machine learning is an iterative process, and it continuously updates its internal model to adapt to new data, thereby achieving more accurate data anomaly detection.

[0145] S330, analyzing the data based on the abnormality score obtained in the comprehensive data calculation module.

[0146] In this step, the established model can be used to evaluate and judge the anomaly scores of all data points, so as to determine whether each data point is an anomaly. Specifically, the threshold can be set to determine which scores are abnormal scores and which scores are normal scores. For example, when the anomaly score is greater than a certain set threshold, the data point is classified as an anomaly. Therefore, this algorithm can be used in anomaly detection problems to find potential abnormal data points by judging and classifying the abnormal scores of data points.

[0147] Figure 8 The flow chart of the embodiment of the present invention provides a real-time monitoring model establishment module evaluation result and establishes an early warning indicator, and compares the evaluation result of the model establishment module with the early warning indicator, such as Figure 8 As shown, the evaluation results of the real-time monitoring model building module and the establishment of early warning indicators, and the comparison of the evaluation results of the model building module with the early warning indicators, include the following steps:

[0148] S410, establishing data early warning indicators in combination with the learning data of the model building module;

[0149] In step S410, the step of establishing data early warning indicators in combination with the learning data of the model building module includes the following sub-steps:

[0150] Step S4101, obtaining a learning data set based on the learning data of the model building module, and confirming the mean value and the standard deviation of the learning data set;

[0151] Step S4102, obtaining the absolute value of the skewness and the absolute value of the kurtosis of the learning data set;

[0152] Step S4103, calculating the data warning index according to the mean, standard deviation, absolute value of skewness and absolute value of kurtosis of the learning data set.

[0153] Among them, the calculation formula of the data warning indicator is expressed as:

[0154]

[0155] Among them, W represents the data warning indicator, δ represents the mean of the learning data set, ε represents the first adjustment coefficient, φ represents the standard deviation of the learning data set, Max represents the maximum value in the learning data set, Min represents the minimum value in the learning data set, |Skewness| represents the absolute value of the skewness of the learning data set, and |Kurtosis| represents the absolute value of the kurtosis of the learning data set.

[0156] S420, comparing the data analysis results obtained by the model building module with the data warning index, and when the analysis results exceed the data warning index, marking the data and issuing a warning notification;

[0157] In this step, according to the model analysis results, the corresponding threshold is set, and the warning mechanism is triggered when the abnormal data exceeds the set threshold. The warning information can be implemented in a variety of ways, such as SMS, email, instant messaging, etc.

[0158] S430, organize the monitoring and early warning results, and optimize the algorithms and models.

[0159] In this step, the thresholds and models of monitoring and early warning need to be regularly updated and adjusted to ensure the effectiveness and reliability of the monitoring mechanism. At the same time, the results of monitoring and early warning should be fed back and sorted out in a timely manner, and the algorithms and models should be optimized to prepare for the next round of monitoring and early warning.

[0160] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0161] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0162] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0163] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

[0164] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A big data platform management method, It is characterized in that The method comprises the following steps: Clean and filter the data to filter out erroneous or invalid data; Select an algorithm and calculate each potential abnormal data to obtain an abnormal score of the potential abnormal data; Establish a data analysis model and evaluate the data status based on the anomaly score obtained by the data calculation module; Monitor the evaluation results of the model building module in real time and establish early warning indicators. Compare the evaluation results of the model building module with the early warning indicators, and issue data warnings when data exceeding the indicators appears; The cleaning and screening of data to filter out erroneous or invalid data includes the following steps: Remove and repair missing, duplicate, outlier, and erroneous values ​​from the collected data; According to the current storage environment, data in different formats are uniformly converted; Combine data from different sources into one data set and merge and remove duplicate fields; Reduce the amount of data stored and the processing time while retaining the main features of the data; Select features related to the analysis target from a large amount of data to improve the accuracy of modeling.

2. A big data platform management method according to claim 1, It is characterized in that In the step of selecting features related to the analysis target from a large amount of data to improve the accuracy of modeling, a feature selection function is used; The feature selection function corresponds to the following function expression: Among them, F(X) represents the output of the feature selection function, which is used to quantify the relevance of the entire feature set to the analysis target; X represents the set of input data, that is, the feature vector; n represents the number of features in the feature set, i represents the feature sequence number, ω i Represents feature x i The weight, T i Represents feature x i Correlation coefficient with the target, μ i Represents feature x i The average value of i Represents feature x i The standard deviation is used to measure the dispersion of the data.

3. A big data platform management method according to claim 2, It is characterized in that The step of selecting an algorithm and calculating each potential abnormal data to obtain the abnormality score of the potential abnormal data includes the following sub-steps: Confirm each potential abnormal data and obtain the features contained in each potential abnormal data; Obtain the feature value corresponding to each feature among all the features contained in each potential abnormal data; The anomaly score of the potential abnormal data is calculated based on the feature value corresponding to each feature of all features contained in the potential abnormal data, the average value of the feature, and the standard deviation of the feature; Among them, the function expression corresponding to the anomaly score of potential abnormal data is: d=(f 1 ,f 2 ,...,f i ...,f m ) Among them, A(d) represents the anomaly score of potential anomaly data d, f i represents the feature value of the i-th feature of the potential abnormal data d, and m represents the number of features contained in the potential abnormal data d.

4. A big data platform management method according to claim 3, It is characterized in that The establishment of the data analysis model and the evaluation of the data status in combination with the abnormality score obtained by the data calculation module include the following steps: Establish models according to the form of data, and optimize and verify the established models; Artificial intelligence and machine learning algorithms are used to train the established model in combination with the anomaly scores obtained in the data calculation module; The data is analyzed using anomaly scores obtained in the comprehensive data calculation module.

5. A big data platform management method according to claim 4, It is characterized in that In the step of training the established model using artificial intelligence and machine learning algorithms and combining the anomaly scores obtained in the data calculation module, the training is performed through a loss function, which takes the anomaly scores as input and is used to optimize model parameters; Among them, the functional expression of the loss function is: Among them, L(θ, X, y) represents the loss function, f θ (x k ) represents the model's response to the input data x k The predicted value, y represents the actual target value, x k represents the kth input data point, y k Represents x k The corresponding true target value, N represents the number of samples in the data set, A(x k ) represents the input data point x k anomaly score, θ represents the parameter set of the model; α represents the first positive hyperparameter, which is used to control the weight of the anomaly score item in the loss function; β represents the second positive hyperparameter, which is used to control the nonlinear effect of the anomaly score on the loss function.

6. A big data platform management method according to claim 5, It is characterized in that The evaluation results of the real-time monitoring model building module and the establishment of early warning indicators, and the comparison of the evaluation results of the model building module with the early warning indicators, include the following steps: Combine the learning data of the model building module to establish data warning indicators; Compare the data analysis results obtained by the model building module with the data warning indicators. When the analysis results exceed the data warning indicators, mark the data and issue a warning notification; Organize the results of monitoring and early warning, and optimize the algorithms and models.

7. A big data platform management method according to claim 6, It is characterized in that Combined with the learning data of the model building module, the steps of establishing data early warning indicators include the following sub-steps: Obtaining a learning data set based on the learning data of the model building module, and confirming the mean value and the standard deviation of the learning data set; Obtain the absolute value of the skewness and kurtosis of the learning data set; According to the mean, standard deviation, absolute value of skewness and absolute value of kurtosis of the learning data set, the data warning indicators are calculated; Among them, the calculation formula of the data warning indicator is expressed as: Among them, W represents the data warning indicator, δ represents the mean of the learning data set, ε represents the first adjustment coefficient, φ represents the standard deviation of the learning data set, Max represents the maximum value in the learning data set, Min represents the minimum value in the learning data set, |Skewness| represents the absolute value of the skewness of the learning data set, and |Kurtosis| represents the absolute value of the kurtosis of the learning data set.

8. A big data platform management system, It is characterized in that The big data platform management method according to any one of claims 1 to 7 is applied, wherein the system comprises: The preprocessing module is used to clean and filter the data and filter out erroneous or invalid data; A data calculation module is used to select an algorithm and calculate each potential abnormal data to obtain an abnormal score of the potential abnormal data; The model building module is used to build a data analysis model and evaluate the data status in combination with the abnormality score obtained by the data calculation module; The monitoring and early warning module is used to monitor the evaluation results of the model building module in real time and establish early warning indicators, compare the evaluation results of the model building module with the early warning indicators, and issue data warnings when data exceeding the indicators appears.

9. A big data platform management system according to claim 8, It is characterized in that The preprocessing module comprises: Data cleaning unit, used to remove and repair missing values, duplicate values, abnormal values ​​and erroneous values ​​of the collected data; A data conversion unit, used to uniformly convert data in different formats according to the current storage environment; Data integration unit, which is used to merge data from different sources into one data set and merge and remove duplicate fields; Data reduction to simple elements is used to reduce the amount of data storage and processing time while retaining the main features of the data; The feature selection unit is used to select features related to the analysis target from a large amount of data to improve the accuracy of modeling.

10. A big data platform management system according to claim 9, It is characterized in that The model building module includes: A model building unit is used to build a model according to the form of data, and optimize and verify the built model; A machine learning unit, for training the established model using artificial intelligence and machine learning algorithms in combination with the anomaly scores obtained in the data calculation module; A data analysis unit, used to analyze the data based on the abnormality scores obtained in the comprehensive data calculation module; The monitoring and early warning module includes: The indicator establishment unit is used to establish data early warning indicators in combination with the learning data of the model establishment module; The data comparison unit is used to compare the data analysis results obtained by the model building module with the data warning index. When the analysis results exceed the data warning index, the data is marked and a warning notification is issued; The module optimization unit is used to organize the monitoring and early warning results and optimize the algorithms and models.

Citation Information

Patent Citations

  • Characteristic selection stability improvement method oriented to performance prediction and ensemble learning

    CN117076897A

  • Application software anomaly detection method and system applying artificial intelligence and storage medium

    CN118277251A

Cited By

  • Clearing data anomaly detection method, device and equipment and medium

    CN119442117A