Index monitoring and early warning method based on big data

Abstract

The invention belongs to the field of data index monitoring and early warning, and particularly relates to an index monitoring and early warning method based on big data, which comprises the following steps: step 1, preprocessing data, including data acquisition and data storage; 2, data abnormal values are eliminated, abnormal detection is carried out on the data through a local outlier factor algorithm in machine learning, a mean value point of historical data is introduced as a reference, and abnormal values deviating from normal data distribution are recognized and eliminated; 3, performing early warning identification on the data and abnormal values thereof to obtain an early warning data set S; and 4, performing monitoring and early warning on the early warning data set S.

Description

Technical Field

[0001] This application belongs to the field of data indicator monitoring and early warning, specifically involving a big data-based indicator monitoring and early warning method. Background Technology

[0002] With the development of technologies such as the Internet and the Internet of Things, enterprises' informatization and intelligent technologies are also constantly developing. The speed and scale of data generation are growing rapidly. This data contains a large amount of valuable information, but it also brings huge challenges. How to efficiently process and analyze this massive amount of data has become one of the core needs.

[0003] In practical data analysis, identifying key data indicators and issuing early warnings is a critical technical requirement. Traditional methods typically rely on simple judgments or statistical analysis. While these methods are effective, they have many limitations when dealing with high-concurrency, multi-source, and heterogeneous big data.

[0004] Enterprise operations often involve collaborative R&D across multiple units, resulting in a complex process that generates semi-structured and structured data. To accelerate the overall R&D progress, this paper explores how to extract the potential value of process data, empower R&D, and achieve monitoring and early warning of key indicators. However, due to the confidentiality of enterprise information management and the physical isolation limitations of the network environment, data indicator monitoring and early warning work faces even greater challenges.

[0005] Currently, data indicator monitoring and early warning mainly employ three categories of methods: rule-based methods, statistical threshold model methods, and machine learning-based anomaly detection methods, as detailed below:

[0006] Rule-based approach: In traditional data metric monitoring and early warning methods, monitoring rules are usually predefined manually and implemented through preset thresholds. For example, a monitoring system might set fixed thresholds based on historical experience or business needs, such as a process backlog exceeding 5 days, and issue an alarm when the metric reaches or exceeds these thresholds. This method is simple, intuitive, easy to implement, and suitable for many scenarios.

[0007] Statistical threshold model-based methods: To address the issue of fixed rules, some data indicator monitoring and early warning methods introduce statistical threshold models. These methods typically analyze historical data of the indicator, calculate the mean, variance, or other statistical characteristics, and dynamically adjust the monitoring threshold based on the statistical distribution. For example, if an indicator exceeds 1.5 times its historical mean, it is considered abnormal. This method is more flexible than rule-based methods and is suitable for more flexible scenarios.

[0008] Machine learning-based anomaly detection methods: With the development of machine learning technology, some data indicator monitoring and early warning methods have begun to adopt machine learning-based anomaly detection methods. These methods train models using historical data, such as clustering, classification, and time series prediction. These methods are generally divided into unsupervised learning methods and supervised learning methods. This approach can discover complex anomaly patterns. For example, it can use a clustering algorithm to group indicator data, marking data points that deviate from normal clusters as anomalies and providing monitoring and early warning.

[0009] Current data indicator monitoring and early warning methods have several limitations. They rely on fixed rules, making them unable to adapt to business changes and slow to react to anomalies. Their monitoring scope is limited, failing to cover all key indicators and potentially creating blind spots. When faced with sudden anomalies, existing systems often only trigger warnings when indicators have already significantly deviated from normal ranges, missing early intervention opportunities. They also suffer from low processing efficiency, lacking real-time data analysis capabilities in big data environments, impacting processing efficiency. Furthermore, they lack adaptability, requiring frequent rule adjustments for different business scenarios. The specific shortcomings of the three types of key data indicator monitoring and early warning methods mentioned above are as follows:

[0010] Rule-based approach: As business grows rapidly, monitoring and early warning needs may change, and fixed monitoring and early warning rules cannot adapt to such dynamic changes; enterprises have a massive number of indicators, and if the rules for manually defining indicators are too labor-intensive, it is difficult to cover all possible anomalies; fixed rules cannot detect potential abnormal trends in a timely manner, and early warnings are only triggered when indicators deviate significantly from the normal range, which may lead to the problem being discovered only after it has worsened.

[0011] Methods based on statistical threshold models: Since many statistical methods assume that the data follows a normal distribution or other specific distribution, but in actual business scenarios, the data may have asymmetry or long-tail characteristics, resulting in poor prediction performance of statistical threshold models; Statistical threshold models are very sensitive to outliers, and the presence of outliers may affect the accuracy of the model and lead to false warnings; The actual operating environment and data distribution of the business may change over time, and traditional statistical threshold models are difficult to adapt to these rapid changes.

[0012] Traditional machine learning-based anomaly detection methods have several drawbacks: training machine learning models requires a large amount of historical data and computing resources, and the models need to be updated frequently as business changes occur, increasing maintenance costs; machine learning methods are more suitable for offline data analysis and have significant limitations for real-time monitoring scenarios; many machine learning models are black-box characteristics, lacking interpretability and making it difficult to detect the specific causes of anomalies, which brings difficulties to troubleshooting.

[0013] Rule-based methods for indicator early warning are simpler and more intuitive, but require a lot of manual intervention and cannot guarantee timely correction of indicator changes caused by business changes. Statistical threshold model-based methods can combine historical data for monitoring and early warning, making indicator formation more evidence-based, but they have limitations due to indicator bias caused by abnormal data and prediction bias caused by assumed data distribution. Machine learning-based anomaly detection methods are effective, but require high-quality data and have high model training costs. It is difficult to explain or modify the causes of anomalies in black-box models, and different models are suitable for different data types. Model selection needs to be carried out according to different data scenarios, and is greatly limited by the model.

[0014] In conclusion, considering the actual business scenarios, effectively monitoring and alerting on data metrics requires a comprehensive approach that integrates the aforementioned methods. Adopting a single technical approach cannot perfectly adapt to real-world work situations. Therefore, this application is submitted. Summary of the Invention

[0015] The purpose of this application is to provide a big data-based indicator monitoring and early warning method to effectively obtain data indicators and provide more reliable data for monitoring and early warning work.

[0016] The technical solution of this application is:

[0017] A big data-based indicator monitoring and early warning method includes:

[0018] Step 1: Preprocess the data, including data acquisition and data storage;

[0019] Step 2: Remove outliers. The local outlier factor algorithm in machine learning is used to detect anomalies in the data. The mean of historical data is used as a reference to identify and remove outliers that deviate from the normal data distribution.

[0020] Step 3: Identify early warnings for the data and its outliers to obtain the early warning dataset S;

[0021] Step 4: Monitor and issue warnings for the early warning dataset S.

[0022] According to at least one embodiment of this application, in the above-described big data-based indicator monitoring and early warning method, step one specifically includes:

[0023] Step 11, Data Collection:

[0024] It relies on in-house data collection tools to extract the required data from the data source system;

[0025] Step 12, Data Storage:

[0026] Based on the enterprise's internal data warehouse, the data warehouse design concept follows the dimensional modeling idea, dividing the data into the basic data layer STAGE, the common dimension model layer CDM, and the application data layer ADS. Among them, the common dimension model layer CDM further includes the detailed data layer DWD and the summary data layer DWS.

[0027] The basic data layer STAGE is used to synchronize data, directly storing data from the data source system in the data warehouse, and saving or cleaning historical data according to data business needs;

[0028] The Common Dimension Model (CDM) layer processes the data from the STAGE layer (basic data layer) to generate detailed fact data and dimension table data, and then processes the detailed fact data and dimension table data to generate summary data of common indicators.

[0029] After the Common Dimension Model (CDM) layer processes the data, it stores the data in the Application Data Layer (ADS).

[0030] According to at least one embodiment of this application, in the above-described big data-based indicator monitoring and early warning method, step two specifically includes:

[0031] Step 21: Calculate the distance between each data point p and all other data points, sort them from nearest to farthest, and filter out the k nearest neighbors, denoted as N. k (p), calculate the relationship between data point p and N k The reach-disk distance between data points o in (p) k (p,o):

[0032] reach-disk k (p,o)=max(dist(p,o),k-distance(o));

[0033] in,

[0034] dist(p,o) is the distance between data point p and data point o;

[0035] k-distance(o) is the reachability distance between data point o and its k-th nearest neighbor;

[0036] Step 22: Calculate the relationship between data point o and the historical mean point X. avg reach-disk k (avg,o):

[0037] reach-disk k (avg,o)=max(dist(avg,o),k-distance(o));

[0038] in,

[0039] dist(avg,o) represents the distance between data point o and the historical mean point X. avg The distance between them;

[0040] Step 23: Introduce the historical mean point X avg As a reference, weights are assigned, and the local reachability density (LRD) of data point p is calculated. k (p):

[0041]

[0042] α1 + α2 = 1,

[0043] in,

[0044] |N k (p)| is N k (p) The number of data points within the range;

[0045] α1 and α2 are the calculation weights;

[0046] Step 24: Calculate the Local Outlier Factor (LOF) of data point p. k (p):

[0047]

[0048] in,

[0049] LRD k (o) represents the local reachability density of data point o, which can be calculated by referring to the local reachability density (LRD) of data point p. k (p);

[0050] Step 24, if the local outlier LOF of data point p is... k If (p) is close to 1, keep data point p; if the local outlier LOF of data point p is... k If (p) is significantly greater than 1, the data point p is removed.

[0051] According to at least one embodiment of this application, in the above-described big data-based indicator monitoring and early warning method, step three specifically includes:

[0052] Step 31: Based on the actual needs of the staff, set the first warning threshold Rule1 and identify the first warning dataset S1;

[0053] Step 32: Based on historical data, set the second early warning threshold Rule2 and identify the second early warning dataset S2;

[0054] Step 33: Summarize the outlier data and compare it with the first warning threshold Rule1 and the second warning threshold Rule2. Outliers within a 20% fluctuation range are considered reasonable. Identify unreasonable outliers to obtain the third warning dataset S3.

[0055] Step 34: Take the union of the first early warning dataset S1, the second early warning dataset S2, and the third early warning dataset S3 to obtain the early warning dataset S.

[0056] According to at least one embodiment of this application, in the above-described big data-based indicator monitoring and early warning method, in step four, the early warning dataset S is stored in the application data layer (ADS) of the data warehouse.

[0057] The specific implementation method of monitoring and early warning is to use corporate email as a carrier and call the email's SMTP service to achieve point-to-point precise monitoring and early warning;

[0058] By attaching specific page links in the email, users can access their personal workbench with one click to view all unfinished tasks, including those that have not met warning indicators.

[0059] Design a backend monitoring visualization module to record information related to the sender, recipient, time, and status of emails. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the big data-based indicator monitoring and early warning method provided in the embodiments of this application;

[0061] Figure 2 This is a schematic diagram of the overall architecture of the data warehouse provided in the embodiments of this application;

[0062] Figure 3 This is a schematic diagram illustrating the data warehouse construction process provided in this application embodiment.

[0063] To better illustrate this embodiment, some content in the accompanying drawings may be omitted, enlarged, or reduced. They are for illustrative purposes only and should not be construed as limiting the scope of this application. Detailed Implementation

[0064] To make the technical solution and advantages of this application clearer, the technical solution of this application will be described in a clearer and more complete manner below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only some embodiments of this application, and are only used to explain this application, not to limit this application. It should be noted that, for ease of description, only the parts related to this application are shown in the accompanying drawings, and other related parts can be referred to the general design.

[0065] Furthermore, unless otherwise defined, the technical or scientific terms used in this application description shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The word "comprising" as used in this application description indicates that the concept preceding the word encompasses the concepts listed following the word and their equivalents, without excluding other related concepts.

[0066] A big data-based indicator monitoring and early warning method, such as Figure 1 As shown, the entire indicator monitoring and early warning process and method have been optimized and improved through several modules, including data preprocessing, indicator modeling, monitoring and early warning, and visualization, which can achieve effective data cleaning, indicator modeling, monitoring and early warning, and visualization presentation.

[0067] Step 1: Preprocess the data.

[0068] Simple preprocessing of data can provide a higher quality data foundation for subsequent outlier identification, including data collection and data storage.

[0069] Step 11: Data collection.

[0070] Data collection can be carried out using in-house data collection tools to extract the required data from the data source system. The specific data extraction process will not be described in detail here.

[0071] Step 12, Data storage.

[0072] Data storage can be carried out using an in-house data warehouse. The design philosophy of the data warehouse follows the dimensional modeling approach, such as... Figure 2 As shown, the data is divided into the basic data layer STAGE, the common dimension model layer CDM, and the application data layer ADS. The common dimension model layer CDM further includes the detailed data layer DWD and the summary data layer DWS.

[0073] The primary function of the base data layer (STAGE) is to synchronize data, storing data directly from the data source system in the data warehouse, and saving or cleaning historical data according to data business needs.

[0074] If a person's gender is stored in three data source systems A, B, and C, inconsistencies in the data definitions of the tables in the three data source systems will occur when building the data warehouse, as shown below: Data source system A Data source system B Data source system C male male 0

[0075] Data source system A directly saves gender as male, data source system B saves it as male, and data source system C stores it as a code value. At this point, it is necessary to build a unified dimensional model to standardize and unify the gender attributes of different data source systems.

[0076] The Common Dimension Model (CDM) layer processes the data from the STAGE layer (basic data layer) to generate detailed fact data and dimension table data, and then processes the detailed fact data and dimension table data to generate summary data of common indicators.

[0077] Taking gender as an example, standard code values ​​can be constructed, where 0 represents male and 1 represents female. The corresponding dimension table data is shown in the table below: Standard Code System A System B System C 0 male male 0 1 female female 1 1 / girl /

[0078] By using dimension table data, gender-related data from various data source systems can be unified.

[0079] After the Common Dimension Model (CDM) layer processes the data, it stores the data in the Application Data Layer (ADS).

[0080] Step 2: Remove outliers from the data.

[0081] By using the local outlier factor algorithm in machine learning to detect anomalies in the data, outliers that deviate from the normal data distribution are identified and removed, laying the foundation for indicator modeling.

[0082] The Local Outlier Factor (LOF) algorithm is a density-based anomaly detection algorithm that determines the degree of anomaly by comparing the density differences between a data point and its neighbors. This density-based LEF algorithm has fewer requirements on data distribution, making it simpler and more intuitive. Its LEF value reflects the degree of anomaly of a data point relative to its neighbors. Density-based outlier detection assumes that the density around a non-outlier object is similar to the density around its neighborhood, while the density around an outlier object is significantly different from the density around its neighborhood. In this application, in addition to calculating the distance of each data point to all other points, the distance between the historical mean and all points is calculated as a reference. The LEF score of the k-neighbors of the historical mean is calculated and used as a benchmark to verify the LEF scores of all other points. Points with LEF scores higher than the k-neighbors of the historical mean are considered outliers.

[0083] Step 21: Calculate the distance between each data point p and all other data points, sort them from nearest to farthest, and filter out the k nearest neighbors, denoted as N. k (p), calculate the relationship between data point p and N k The reach-disk distance between data points o in (p) k (p,o).

[0084] reach-disk k (p,o)=max(dist(p,o),k-distance(o));

[0085] in,

[0086] dist(p,o) is the distance between data point p and data point o;

[0087] k-distance(o) is the reachability distance between data point o and its k-th nearest neighbor.

[0088] Step 22: Calculate the relationship between data point o and the historical mean point X. avg reach-disk k (avg,o).

[0089] reach-disk k (avg,o)=max(dist(avg,o),k-distance(o));

[0090] in,

[0091] dist(avg,o) represents the distance between data point o and the historical mean point X. avg The distance between them.

[0092] Locally accessible density (LRD) of data point p k (p) is usually defined as N k (p) The reciprocal of the average reachable distance of interior points is calculated as follows:

[0093]

[0094] in,

[0095] |N k (p)| is N k The number of data points in (p) is k.

[0096] Step 23: Introduce the historical mean point X avg As a reference, weights are assigned, and the local reachability density (LRD) of data point p is calculated. k (p).

[0097]

[0098] α1 + α2 = 1,

[0099] in,

[0100] α1 and α2 are the calculation weights. Usually, α1 can be 0.8 and α2 can be 0.2.

[0101] Step 24: Calculate the Local Outlier Factor (LOF) of data point p. k (p), defined as N k(p) The average local reachability density of data points within the range, and its local reachability density (LRD) k The ratio of (p) is calculated as follows:

[0102]

[0103] in,

[0104] LRD k (o) represents the local reachability density of data point o, which can be calculated by referring to the local reachability density (LRD) of data point p. k (p).

[0105] Step 24, if the local outlier LOF of data point p is... k If the density of data point p is close to 1, it indicates that the density of data point p is similar to that of its neighboring points. In this case, p is considered not an outlier and the data point p is retained. If the local outlier factor (LOF) of data point p is close to 1, it means that the density of data point p is similar to that of its neighboring points. k If (p) is significantly greater than 1, it indicates that data point p may be an outlier, i.e., an anomaly, and should be removed.

[0106] Step 3: Identify the data and its outliers to generate an early warning dataset S.

[0107] Since enterprises are typically offline, current data collection tools, while achieving near real-time collection, consume excessive scheduling resources. To ensure the optimal and efficient use of enterprise resources, this application performs data synchronization daily, allowing ample time for data calculation and preparation during indicator modeling. Against this backdrop, this application comprehensively considers defined rules, statistical threshold models, and outlier removal through machine learning, combining the indicator data from these three sources to form the final early warning dataset S available for alerts.

[0108] Step 31: Based on the actual needs of the staff, set the first warning threshold Rule1 and identify the first warning dataset S1.

[0109] For example, "If staff members believe that a certain process has accumulated more than 3 days in a certain step, an early warning reminder is needed." Such rules are derived from work experience and, after being quantified, are recorded as the first early warning threshold, Rule1.

[0110] Step 32: Based on historical data, set the second warning threshold Rule2 and identify the second warning dataset S2.

[0111] For example, "The historical completion time of other similar work processes is 20 days. The current process has exceeded 20 days. Such data is considered abnormal data." This type of rule is calculated from historical data and quantified as the second warning threshold Rule2.

[0112] Step 33: Summarize the outlier data and compare it with the first warning threshold Rule1 and the second warning threshold Rule2. Outliers within a 20% fluctuation range are considered reasonable. Identify unreasonable outliers to obtain the third warning dataset S3. Remove excessively large fluctuation data to reduce errors caused by the instability of the machine learning model.

[0113] Step 34: Take the union of the first early warning dataset S1, the second early warning dataset S2, and the third early warning dataset S3 to obtain the early warning dataset S.

[0114] In a specific example, during the entire lifecycle of a company's production of a certain type of product, an approval process is required. This process data includes attributes such as product name, specific category, product owner, sender, receiver, current stage, approver at each stage, arrival time at the current stage, and dwell time. To ensure the process can complete approval quickly and accelerate product production, warnings need to be issued for products with excessively long dwell times. Since the actual approval process involves a large number of personnel, it is necessary to select appropriate warning indicators that are neither offensive to the public nor hinder the approval process.

[0115] First, collect the actual needs of the relevant product managers. Based on their work experience, the relevant product managers give the first warning threshold Rule1, which states that "this process must be completed and approved within 15 days". Data that has exceeded 15 days is recorded as the first warning dataset S1.

[0116] Secondly, based on historical data, the average time of similar processes is calculated, and a second warning threshold Rule2 is given. Data whose process time exceeds the average time of similar processes so far are recorded as the second warning dataset S2.

[0117] Finally, the local outlier factor algorithm is used to calculate the local outlier factor. Data with a value greater than 1 is considered outlier data. Compared with the first warning threshold Rule1 and the second warning threshold Rule2, data with an outlier fluctuation of less than 20% is removed, that is, data whose process has not exceeded 12 days and whose current average time has not reached 80% of the historical average time are removed, resulting in the third warning dataset S3. The union of the first warning dataset S1, the second warning dataset S2, and the third warning dataset S3 is taken to obtain the warning dataset S.

[0118] Step 4: Monitor and issue warnings for the early warning dataset S.

[0119] The early warning dataset S can be stored in the application data layer ADS of the data warehouse.

[0120] Based on user work habits and actual scenario research, most people pay close attention to corporate email. Therefore, corporate email can be chosen as the carrier for monitoring and alerting. Since most enterprises have physical network isolation, this application utilizes the email's SMTP service to achieve precise point-to-point monitoring and alerting.

[0121] After integration with the email system, the relevant personnel will receive [information / received information ... <datacenter>Email alerts and notifications will be provided. To provide a better user experience, specific page links can be attached to the emails, allowing users to directly access their personal workbench to view all unfinished tasks, including those that have not met alert criteria. To effectively monitor the entire service process and ensure administrators have full control over the status, a backend monitoring visualization module will be designed to record information such as email sender, recipient, time, and status.

[0122] The big data-based indicator monitoring and early warning method disclosed in the above embodiments has the following beneficial technical effects compared with the prior art:

[0123] A local outlier factor algorithm is introduced, and the formulas for calculating local reachability density and local outlier factor in the algorithm are optimized. Based on the actual characteristics of the enterprise, historical relevant indicator data, namely historical mean points, are introduced, and weights are allocated when calculating local reachability density, referring to historical data indicators.

[0124] By combining rule-based anomaly detection methods, statistical threshold-based anomaly detection methods, and machine learning-based anomaly detection methods, and taking into account the specific characteristics and actual situation of enterprises, this approach not only meets the actual needs of users and provides a reference for historical data, but also introduces intelligent algorithms to explore data anomalies from new dimensions, ensuring more reliable data results through multi-dimensional consideration.

[0125] Utilize existing tools and methods within the enterprise to ensure users receive information quickly via email push notifications.

[0126] A visual interface is built to organically combine early warning information with an individual's unfinished work. The early warning work serves as a starting point to introduce all unfinished work, empowering the entire work process. A comprehensive monitoring interface is also provided to reasonably monitor the entire process of the early warning business and intuitively obtain all results.

[0127] The technical solution of this application has been described in conjunction with the preferred embodiments shown in the accompanying drawings. Those skilled in the art should understand that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.< / datacenter>

Claims

1. A big data-based indicator monitoring and early warning method, characterized in that, include: Step 1: Preprocess the data, including data acquisition and data storage; Step 2: Remove outliers. The local outlier factor algorithm in machine learning is used to detect anomalies in the data. The mean of historical data is used as a reference to identify and remove outliers that deviate from the normal data distribution. Step 3: Identify early warnings for the data and its outliers to obtain the early warning dataset S; Step 4: Monitor and issue warnings for the early warning dataset S.

2. The big data-based indicator monitoring and early warning method according to claim 1, characterized in that, Step one specifically includes: Step 11, Data Collection: It relies on in-house data collection tools to extract the required data from the data source system; Step 12, Data Storage: Based on the enterprise's internal data warehouse, the data warehouse design concept follows the dimensional modeling idea, dividing the data into the basic data layer STAGE, the common dimension model layer CDM, and the application data layer ADS. Among them, the common dimension model layer CDM further includes the detailed data layer DWD and the summary data layer DWS. The STAGE layer, which is the basic data layer, is used to synchronize data. It stores data from the data source system directly in the data warehouse and saves or cleans historical data according to data business needs. The Common Dimension Model (CDM) layer processes the data from the STAGE layer (basic data layer) to generate detailed fact data and dimension table data, and then processes the detailed fact data and dimension table data to generate summary data of common indicators. After the Common Dimension Model (CDM) layer processes the data, it stores the data in the Application Data Layer (ADS).

3. The big data-based indicator monitoring and early warning method according to claim 2, characterized in that, Step two specifically includes: Step 21: Calculate the distance between each data point p and all other data points, sort them from nearest to farthest, and filter out the k nearest neighbors, denoted as N. k (p), calculate the relationship between data point p and N k The reach-disk distance between data points o in (p) k (p,o): reach-disk k (p,o)=max (dist(p,o),k-distance(o)); in, dist(p,o) is the distance between data point p and data point o; k-distance(o) is the reachability distance between data point o and its k-th nearest neighbor; Step 22: Calculate the relationship between data point o and the historical mean point X. avg reach-disk k (avg,o): reach-disk k (avg,o)=max(dist(avg,o),k-distance(o)); in, dist(avg,o) represents the distance between data point o and the historical mean point X. avg The distance between them; Step 23: Introduce the historical mean point X avg As a reference, weights are assigned, and the local reachability density (LRD) of data point p is calculated. k (p): α1+α2＝1， in, |N k (p)| is N k (p) The number of data points within the range; α1 and α2 are the calculation weights; Step 24: Calculate the Local Outlier Factor (LOF) of data point p. k (p): in, LRD k (o) represents the local reachability density of data point o, which can be calculated by referring to the local reachability density (LRD) of data point p. k (p); Step 24, if the local outlier LOF of data point p is... k If (p) is close to 1, keep data point p; if the local outlier LOF of data point p is... k If (p) is significantly greater than 1, the data point p is removed.

4. The big data-based indicator monitoring and early warning method according to claim 3, characterized in that, Step three specifically includes: Step 31: Based on the actual needs of the staff, set the first warning threshold Rule1 and identify the first warning dataset S1; Step 32: Based on historical data, set the second early warning threshold Rule2 and identify the second early warning dataset S2; Step 33: Summarize the outlier data and compare it with the first warning threshold Rule1 and the second warning threshold Rule2. Outliers within a 20% fluctuation range are considered reasonable. Identify unreasonable outliers to obtain the third warning dataset S3. Step 34: Take the union of the first early warning dataset S1, the second early warning dataset S2, and the third early warning dataset S3 to obtain the early warning dataset S.

5. The big data-based indicator monitoring and early warning method according to claim 4, characterized in that, In step four, the early warning dataset S is stored in the application data layer (ADS) of the data warehouse; The specific implementation method of monitoring and early warning is to use corporate email as a carrier and call the email's SMTP service to achieve point-to-point precise monitoring and early warning; By attaching specific page links in the email, users can access their personal workbench with one click to view all unfinished tasks, including those that have not met warning indicators. Design a backend monitoring visualization module to record information related to the sender, recipient, time, and status of emails.

Images