Historical local anomaly-based data evaluation method, apparatus and device, and medium

By combining the LOF algorithm with the Z-score method for data evaluation, the problem of low efficiency of manual evaluation in environmental monitoring is solved, data anomalies can be quickly identified, and data accuracy and the precision of governance strategies are improved.

CN120611304APending Publication Date: 2025-09-09INST OF URBAN ENVIRONMENT CHINESE ACAD OF SCI +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202510639056.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-05-19
Publication Date
2025-09-09

AI Technical Summary

Technical Problem

Existing technologies in environmental monitoring rely on manual experience to assess data quality, which is inefficient and susceptible to subjective bias. It is difficult to quickly discover anomalies and abnormal patterns in multi-dimensional data. Especially when faced with large-scale and complex data, traditional methods are unable to meet the needs of efficient and accurate data quality assessment.

Method used

A data evaluation method based on historical local anomalies is adopted, combined with the LOF algorithm and the Z-score method. By constructing a reference density distribution, historical data is used to measure the local density differences of new data, the degree of outliers of sample points is quantified, and dimension-by-dimensional analysis is performed to identify specific anomaly indicators.

Benefits of technology

It has achieved efficient and reliable evaluation of environmental monitoring data, can quickly discover potential anomalies, improve data accuracy, provide solid technical support for environmental governance decisions, and promote intelligent and precise development.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120611304A_ABST
    Figure CN120611304A_ABST
Patent Text Reader

Abstract

The invention provides a data evaluation method and device based on historical local anomalies, equipment and a medium. The method comprises the following steps: performing data preprocessing on historical data in environmental monitoring, calculating a k-neighborhood distance, a reachable distance, a local reachable density and a local anomaly factor, and generating reference density distribution information; processing the reference density distribution information, merging test data points with historical data one by one to calculate an LOF value, screening out potential abnormal samples in combination with a set threshold, and generating potential abnormal sample information; for potential abnormal samples, Z-Score is calculated according to feature dimensions, and abnormal index information of each dimension is generated in combination with a set threshold value; processing the potential abnormal sample information and the abnormal index information of each dimension, marking a sample abnormal state and positioning an abnormal source index, and generating a sample abnormal state mark and abnormal source index positioning information; and processing the sample abnormal state mark and the abnormal source index positioning information to generate abnormal information of the environmental data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing applications, and in particular to a data evaluation method, device, equipment and medium based on historical local anomalies. Background Art

[0002] In ecological and environmental monitoring, data on water quality and river and lake sedimentation are crucial for assessing environmental status and formulating remediation measures. However, ensuring the accuracy and reliability of this data remains a key challenge. Currently, many monitoring agencies still rely primarily on manual experience and repeated testing to assess data quality. This traditional approach is not only inefficient but also requires significant human and material resources. Furthermore, manual judgment is susceptible to subjective bias and environmental factors, such as insufficient experience or a lack of comprehensive understanding of the sample collection scenario.

[0003] Furthermore, despite increased efforts to improve data reliability, uncontrollable factors such as occasional equipment failures can still lead to discrepancies in parallel sample testing results, which can be difficult to detect in a timely manner. As the scale and complexity of environmental monitoring data continue to grow, relying solely on traditional manual methods is no longer sufficient to efficiently and accurately assess data quality. This is especially true when dealing with multi-dimensional data, making it even more difficult to quickly identify anomalies and patterns.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore includes information that does not constitute prior art known to ordinary technicians in this field. Summary of the Invention

[0005] The purpose of this application is to provide a data evaluation method, device, equipment and medium based on historical local anomalies, which at least to some extent overcomes the problems existing in the prior art. By using a data evaluation method based on the fusion of historical local anomalies and standard scores, the LOF algorithm and the Z score method are integrated. In environmental monitoring, historical data is used to construct a reference density distribution to find potential anomalies, and then the Z score method is used to identify anomaly indicators, providing a new way to evaluate environmental monitoring data, support environmental governance decision-making, and promote the intelligence and precision of related work.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0007] According to one aspect of the present application, a data evaluation method based on historical local anomalies is provided, including: obtaining historical data and target test data in environmental monitoring; performing data preprocessing on the historical data in environmental monitoring, calculating the k-neighborhood distance, reachable distance, local reachable density, and local anomaly factor, and generating reference density distribution information; processing the reference density distribution information, merging the test data points with the historical data one by one to calculate the LOF value, and screening out potential abnormal samples in combination with a set threshold to generate potential abnormal sample information; for the potential abnormal samples, calculating the Z-Score according to the feature dimension, and generating abnormal indicator information of each dimension in combination with a set threshold; processing the potential abnormal sample information and the abnormal indicator information of each dimension, marking the abnormal state of the sample and locating the abnormal source indicator, and generating the sample abnormal state mark and abnormal source indicator location information; processing the sample abnormal state mark and the abnormal source indicator location information to generate abnormal information of the environmental data.

[0008] Another aspect of the present application is a data evaluation device based on historical local anomalies, characterized in that it includes: an acquisition module for acquiring historical data and target test data in environmental monitoring; a processing module for performing data preprocessing on the historical data in environmental monitoring, calculating the k-neighborhood distance, reachable distance, local reachable density, and local anomaly factor, and generating reference density distribution information; processing the reference density distribution information, merging the test data points with the historical data one by one to calculate the LOF value, and screening out potential abnormal samples in combination with a set threshold to generate potential abnormal sample information; calculating the Z-Score of the potential abnormal samples according to the feature dimension, and generating abnormal index information of each dimension in combination with a set threshold; processing the potential abnormal sample information and the abnormal index information of each dimension, marking the abnormal state of the sample and locating the abnormal source indicator, generating the sample abnormal state mark and the abnormal source indicator positioning information; processing the sample abnormal state mark and the abnormal source indicator positioning information to generate abnormal information of the environmental data.

[0009] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a second processor, the computer program implements the above-mentioned data evaluation method based on historical local anomalies.

[0010] This application provides a data evaluation method, apparatus, device, and medium based on historical local anomalies. This data evaluation method, based on the fusion of historical local anomalies and standard scores, innovatively integrates the Local Outlier Factor (LOF) algorithm and the Z-score method. In environmental monitoring data processing, this method uses historical data to construct a reference density distribution. The LOF algorithm measures the local density difference between new and historical data, quantifying the degree of outliers in sample points and rapidly identifying potential anomalies. Subsequently, the Z-score method is used to perform a dimension-by-dimensional analysis of the initially identified potential anomaly samples to accurately identify specific anomaly indicators.

[0011] The LOF-ZFusion model offers significant advantages over traditional anomaly detection methods. It not only deeply mines subtle anomaly information from local data features, but also uses statistical methods to comprehensively quantify anomaly significance, ensuring that detection results are scientific and stable at both the overall and indicator levels. This model provides an efficient and reliable new approach to environmental monitoring data quality assessment, helping to promptly identify data issues and improve data accuracy. It also provides solid technical support for environmental governance decision-making, enabling relevant departments to develop more effective governance strategies based on accurate data, and promoting the intelligent and precise development of environmental monitoring and governance.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A flowchart of a data evaluation method based on historical local anomalies provided by an embodiment of the present application is shown; Figure 2 A schematic structural diagram of a data evaluation device based on historical local anomalies provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0014] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0015] In one embodiment, the present application also proposes a data evaluation method, device, equipment and medium based on historical local anomalies. Figure 1 The following schematically shows a flow chart of a data evaluation method based on historical local anomalies according to an embodiment of the present application. Figure 1 As shown, the method is applied to the server and includes: S101, obtaining historical data and target test data in environmental monitoring.

[0016] In one embodiment, historical data used in environmental monitoring primarily derives from the extensive data accumulated by environmental monitoring agencies through long-term monitoring efforts. This data encompasses various environmental monitoring indicators from different times and locations. For example, in ecological and environmental monitoring, this includes water quality monitoring indicators such as pH, dissolved oxygen, and chemical oxygen demand; as well as river and lake sediment monitoring data, such as heavy metal content and organic matter content. This data is collected through standardized monitoring processes using professional monitoring equipment and undergoes multiple tests and verifications to ensure its accuracy and reliability. Target test data is data collected during the current monitoring process for specific monitoring tasks and objectives. It maintains consistency with historical data in terms of monitoring indicators, facilitating comparative analysis. For example, when conducting real-time monitoring of a water body in a specific area, newly collected water sample test data is the target test data, reflecting the current actual condition of the water body in that area.

[0017] Data must be in numerical format to ensure consistency and operability in subsequent calculations and analyses. Furthermore, data units for the same attribute must be consistent to avoid data errors and analytical bias caused by unit differences. For example, when monitoring the concentration of a specific pollutant in water, all data should be in standard units such as mg / L or μg / L. Missing items should be avoided. If undetected data exist in the test data, they should be replaced with a reasonable value below the detection limit. However, the percentage of undetected data in each attribute should be kept to a minimum (less than 10%). If a small amount of missing data exists (less than 3%), it can be supplemented with the average value of the attribute to ensure data integrity and enhance the reliability of subsequent analysis results. Attribute names in historical data and target test data must be identical to ensure accurate comparison and calculation of the same attribute data in subsequent analysis. For example, if the attribute "dissolved oxygen" exists in both historical and target test data, its name and meaning should be identical in both to ensure data comparability.

[0018] Data collection should be conducted using professional monitoring equipment and scientific sampling methods in accordance with relevant standards and specifications for environmental monitoring. When collecting water samples, multiple sampling points should be conducted at different depths and locations within the water body to ensure that the collected samples are representative of the actual conditions of the water in the area. After collection, the samples should be promptly sent to the laboratory for testing and analysis to obtain the corresponding data. The collected data needs to be organized and entered before entering the evaluation process. A preliminary inspection should be conducted on the collected data to eliminate any obvious errors, and then the data should be entered into a dedicated database or data storage system in a unified format. During the data entry process, the accuracy of the data should be carefully checked to avoid entry errors. The entered data should be properly stored, and a comprehensive data management system should be established. This system should include functions such as data backup, data update, and data retrieval to facilitate the subsequent access and management of historical data and target test data, providing efficient data support for data evaluation.

[0019] S102 , performing data preprocessing on historical data in environmental monitoring, calculating k-neighborhood distance, reachable distance, local reachable density, and local anomaly factor, and generating reference density distribution information.

[0020] In one implementation, historical environmental monitoring data is cleaned to remove erroneous values, duplicate values, and invalid data, generating cleaned historical data. For example, consider historical monitoring data on dissolved oxygen levels in a river. Some records contain negative values ​​(dissolved oxygen levels cannot be negative, so they are erroneous values), as well as duplicate records that are identical. During the cleansing process, these erroneous and duplicate values ​​are removed to generate cleaned historical data. For example, the original data may contain [2.5, -1.0, 2.5, 3.0], which becomes [2.5, 3.0] after cleaning.

[0021] Standardize the cleaned historical data information to generate standardized historical data information; use the formula Standardization is performed, where Is the specific feature value of a sample in the original data, O is the mean of the feature in the historical data, and T is the standard deviation of the feature in the historical data. There is a set of data about chemical oxygen demand (COD) in rivers [10, 20, 30, 40, 50], whose mean O is 30 and standard deviation T is about 14.14. Taking data point 20 as an example, the standardized value , perform similar calculations on all data points to obtain standardized historical data information.

[0022] For a normalized data point set, k=3. For a data point p, calculate its distance to other points in the dataset (e.g., Euclidean distance), then sort them from smallest to largest. The distance to the third nearest neighbor is the k-neighborhood distance (k-distance(p)). For example, if the data point set is [1.2, 2.5, 3.1, 4.7, 5.3], for point 2.5, calculate its distance to the other points, sort them, and take the third distance value as the k-neighborhood distance, thus generating k-neighborhood distance information.

[0023] Based on the k-neighborhood distance information and the standardized historical data information, the formula , calculate the reachable distance of data point p relative to q and generate reachable distance information, where, Represents the distance between point p and point q. , ,but For all data points By performing such calculations, we can obtain the reachable distance information. For example, in a two-dimensional data set, there are points p(1,1) and q(2,2). Calculate 3, the Euclidean distance between two points ,but .

[0024] According to the reachable distance information and the standardized historical data information, the formula , calculate the local reachability density of the sample point p and generate local reachability density information, where, To represent the local reachability density of sample point p, p is the sample point for which the local reachability density is currently being calculated, q is the other data points in the k-neighborhood of point p, is the set of k data points closest to point p in the data set, is the distance between point p and point q. Let the k neighborhood of point p be There are 3 points 、 、 , and the corresponding reachable distances are , , ,but . ,so A similar calculation is performed on each sample point to generate local reachability density information.

[0025] Based on the local reachability density information, using the formula , calculate the local anomaly factor of the sample point p and generate local anomaly factor information, where, is the local anomaly factor of the sample point p, which is used to evaluate whether the point p is an anomaly point. is the local reachability density of sample point q. Assume that there are 3 points in the k-neighborhood of point p 、 、 , =0.33, , , ,but , ,so When the LOF value is approximately 1, the sample density is close to that of the neighborhood. When it is greater than 1, the density of the point is sparser than the neighborhood density and may be an outlier. Calculate the LOF value of all sample points to generate local outlier factor information.

[0026] Process the k-neighborhood distance information, reachable distance information, local reachable density information, and local anomaly factor information to generate reference density distribution information. Set a unique identifier for each data point to facilitate subsequent information matching and management. In water quality monitoring data, if the monitoring time and monitoring location are used as joint identifiers, "20240101-Site A" can clearly represent the water sample data point collected at Site A on January 1, 2024. Build a table framework: Create a table containing data point identifiers and columns such as k-neighborhood distance, reachable distance, local reachable density, and local anomaly factor. The table headers are "Data Point Identifier", "k-Neighborhood Distance", "Reachable Distance", "Local Reachable Density", and "Local Anomaly Factor". Fill in data: Based on the previous calculation results, accurately fill in the table with all kinds of information corresponding to each data point. For the data point "20240101-Site A," assume its k-neighborhood distance is calculated to be a difference of 0.5 mg / L. Fill that value into the corresponding cell. If the calculated reachable distance to a point q in the neighborhood is 0.8, also fill in the corresponding position. If the calculated local reachable density is 2.5 per unit volume, and the local anomaly factor is 1.2, fill in the corresponding cells as well. Repeat this process for all data points.

[0027] Perform a thorough check of the data entered into the table to verify its accuracy and completeness. Check for missing data or significantly abnormal values. If the local reachability density value for a data point is 0, recheck the calculation process or the original data and make corrections. To ensure comparability between different types of data, standardize or normalize some of the data. For k-neighbor distance and reachability distance data, since their numerical ranges and dimensions may differ, standardize them using a standardization formula to map the data to the same scale.

[0028] Analyze the relationship between the local anomaly factor and the local reachable density. If it is found that the points with low local reachable density in certain areas generally have high local anomaly factors, this suggests that there are special circumstances in the data distribution of these areas and they need to be paid special attention. Application in screening potential anomaly samples Compare test data with reference information: After obtaining the test data, for each test data point, find similar data points in the reference density distribution information table (such as based on the principle of similar monitoring location and time in the data point identification). Calculate the differences between the test data point and the corresponding data point in the reference table in terms of k-neighborhood distance, reachable distance, local reachable density and local anomaly factor. Determine the threshold for screening potential anomaly samples. If the local anomaly factor threshold is set to 1.5, when the calculated value of the local anomaly factor of the test data point is greater than 1.5, the test data point is preliminarily marked as a potential anomaly sample. Comprehensively judge potential anomaly samples: not only rely on the local anomaly factor as an indicator, but also consider other information. If the local anomaly factor of a test data point is slightly larger than the threshold, but its k-neighborhood distance is extremely different from the reference data, and the local reachable density also deviates significantly from the reference range, then the test data point is more likely to be a potential anomaly sample; conversely, if only the local anomaly factor slightly exceeds the threshold and other indicators are close to the reference data, further analysis and judgment are required.

[0029] S103, processing the reference density distribution information, combining the test data points with the historical data one by one to calculate the LOF value, combining the set threshold to filter out potential abnormal samples, and generating potential abnormal sample information.

[0030] In one embodiment, the historical data and test data involved in the reference density distribution information are formatted in a unified manner to generate data information in a unified format. The historical data is stored in CSV format, which includes the monitoring time, monitoring location, and various environmental indicators (such as temperature, humidity, pollutant concentration, etc.), and the data is arranged in ascending order of time. The test data is stored in an Excel file, and the data order and some column names are inconsistent with the historical data. When the format is unified, the test data in Excel format is first converted to CSV format, and then the column name and data arrangement order of the test data are adjusted according to the column name order and data type of the historical data. For example, the temperature data column name in the historical data is "Temperature", while in the test data it is "Temp". The column name in the test data is modified to "Temperature", and the data type (such as numeric type) is ensured to be consistent, and finally the data information in a unified format is generated.

[0031] The missing values ​​of the data after formatting are processed. If there are missing temperature data at some monitoring time points in the data set, the mean interpolation method is used for processing. First, the mean temperature in the historical data is calculated, assuming it is 25°C (the calculation method is the sum of all non-missing temperature values ​​in the column divided by the number of non-missing values), and then the missing temperature values ​​in the test data are filled with 25°C. At the same time, the data can also be normalized, and the data of various environmental indicators can be mapped to the [0,1] interval. For example, for temperature data, assuming its maximum value is 40°C and its minimum value is 10°C, for a certain temperature value x, the normalized value , after these processes, processed data information is generated.

[0032] Merging test data with historical data provides a unified dataset for calculating the local outlier factor (LOF) value of the test data point. This allows for assessing the degree of anomaly of the test data point based on the distribution characteristics of the historical data. For example, the processed test data contains data from five monitoring points: T1, T2, T3, T4, and T5. Extract T1's data from the processed data. Assume that T1 contains the monitoring time "2024-01-01 08:00:00", the monitoring location "SiteA", the temperature 28°C, and the humidity 60%. Add T1's data to the historical dataset, which already contains monitoring data from different times and locations. This addition forms a new, merged dataset. Similarly, add T2, T3, T4, and T5 to the historical dataset.

[0033] Based on the merged data set information, according to the local anomaly factor calculation formula , calculate the LOF value of each test data point and generate LOF value information. Let k = 5, for the test data point T1 added to the historical data set, first determine its k neighborhood , that is, the 5 data points closest to T1 (the distance calculation here uses methods such as Euclidean distance, and the distance is calculated based on various indicators). Suppose these 5 data points are 、 、 、 、 , calculate their local reachable density respectively 、 、 、 、 , and the local reachable density LRD(T1) of T1. For example, we can calculate 、 、 、 、 , LRD(T1)=0.2. Then , ,so, According to this method, the LOF value of each test data point is calculated to generate LOF value information.

[0034] The LOF value of each test point is compared with the preset threshold. If the LOF value is greater than the preset threshold, the corresponding test point is marked as a potential anomaly sample, and potential anomaly sample information is generated. The preset threshold is 1.5. The calculated LOF value of T1, 1.9, is compared with the preset threshold of 1.5. Since 1.9>1.5, T1 is marked as a potential anomaly sample. Similarly, the LOF values ​​of other test data points T2, T3, T4, and T5 are compared. Assuming LOF(T2) = 1.2, LOF(T3) = 2.0, LOF(T4) = 1.7, and LOF(T5) = 1.4, T3 and T4 are also marked as potential anomaly samples, while T2 and T5 are not marked. The data information marked as potential anomaly samples is organized together to generate potential anomaly sample information, such as recording the monitoring time, monitoring location, various indicator values, and corresponding LOF values ​​of the potential anomaly samples.

[0035] S104: For potential abnormal samples, calculate the Z-Score according to the feature dimension, and generate abnormal indicator information of each dimension by combining the set threshold.

[0036] In one embodiment, the characteristic dimension information of the potential abnormal sample is extracted and filtered to generate sample data information of each dimension feature. In environmental monitoring data, potential abnormal samples include multiple characteristic dimensions such as temperature, humidity, and pollutant concentration. Taking a potential abnormal sample as an example, it records the environmental data of a certain area at a certain time, including a temperature of 28°C, a humidity of 60%, and a certain pollutant concentration of 50mg / L. From this sample, the temperature dimension data "28°C", the humidity dimension data "60%", the pollutant concentration dimension data "50mg / L", etc. are extracted to generate sample data information of each dimension feature.

[0037] For the sample data information of each dimension feature, the mean and standard deviation of each feature dimension in the historical data are calculated based on the historical data to generate the historical mean information and historical standard deviation information. Taking the above environmental monitoring data as an example, there are 100 data in the temperature dimension in the historical data, and the total is 2500℃. Then the mean value of temperature is To calculate the standard deviation T, first calculate the sum of the squares of the differences between each data point and the mean. If the sum of the squares of the differences is 100, then 1°C. Similarly, the mean and standard deviation of humidity and pollutant concentration in historical data are calculated to generate historical mean information and historical standard deviation information.

[0038] Based on the sample data information, historical mean information and historical standard deviation information of each dimension feature, the Z-Score calculation formula is used , calculate the Z-Score value of each feature dimension and generate the Z-Score value information of each dimension feature, where, For the The Z-Score value of the feature dimension, For potential abnormal samples The sample data value of feature dimensions, For the The mean of feature dimensions in historical data, For the The standard deviation of the feature dimension in historical data.

[0039] For the temperature dimension of the above potential abnormal samples, =28℃, =25℃, =1℃, then . Using the same method, calculate the Z-Score values ​​of humidity and pollutant concentration dimensions, such as humidity dimension =60%, =50%, =5%, then ; Pollutant concentration dimension is set =50mg / L, =30mg / L, =10mg / L, then , generate the Z-Score numerical information of each dimension feature.

[0040] The absolute value of the Z-Score value of each dimension feature is compared with the Z-Score judgment threshold. If the Z-Score absolute value is greater than the set threshold, the feature dimension is marked as abnormal, and abnormal marking information for each dimension is generated. The Z-Score judgment threshold is set to 2. For the temperature dimension, | |=3>2, so the temperature dimension is marked as abnormal; for the humidity dimension, | |=2, not greater than the threshold, not marked as anomaly; for the pollutant concentration dimension, | = 2, not greater than the threshold, not marked as an anomaly. Mark the temperature dimension as an anomaly and generate anomaly marking information for each dimension.

[0041] Summarize the anomaly flag information for each dimension of all potentially abnormal samples. For example, consider five potentially abnormal samples. In addition to the temperature dimension being flagged as abnormal, another sample's pollutant concentration dimension is also flagged as abnormal. This anomaly flag information is organized into a table containing the sample number, the name of each dimension, and whether it is abnormal. This generates anomaly indicators for each dimension, clearly presenting the anomaly status of each dimension in the potentially abnormal samples.

[0042] S105: Process the potential abnormal sample information and abnormal indicator information of each dimension, mark the sample abnormal state and locate the abnormal source indicator, and generate the sample abnormal state mark and abnormal source indicator location information.

[0043] In one embodiment, identification information of the potential abnormal sample is extracted from the potential abnormal sample information. In environmental monitoring data, it is assumed that the identification of each sample consists of the monitoring time, monitoring location, and sample number. For a potential abnormal sample, its identification information may be "2024-09-15 10:00:00-Site A-001", where "2024-09-15 10:00:00" represents the monitoring time, "Site A" is the monitoring location, and "001" is the sample number. Accurately extracting this identification information from the potential abnormal sample information provides a basis for subsequent processing.

[0044] From the anomaly indicator information for each dimension, we extract the anomaly flag information and anomaly dimension information for each potential anomaly sample. After the previous calculations and analysis, we obtain the anomaly flag information for each of the three dimensions of temperature, humidity, and pollutant concentration. The anomaly indicator information for each dimension records the sample's abnormal temperature dimension ("1" indicates abnormal, "0" indicates normal), the humidity dimension (normal), and the pollutant concentration dimension (abnormal). Furthermore, the abnormal dimensions are clearly defined as temperature and pollutant concentration. This information will be used to subsequently determine the sample's abnormal status and locate the source of the anomaly.

[0045] The identification information, anomaly flags for each dimension, and abnormal dimension information of a potential anomaly sample are processed. If at least one dimension of the sample is marked as abnormal, information is generated to determine the sample's abnormal status. Otherwise, the sample is considered normal and the dimension information marked as abnormal is used as the basis for locating the source of the anomaly. For the potential anomaly sample "2024-09-15 10:00:00-Site A-001," since both its temperature and pollutant concentration dimensions are marked as abnormal, meaning at least one dimension is abnormal, information is generated to determine the sample's abnormal status and determine it is an anomaly. Furthermore, the two dimensions marked as abnormal, temperature and pollutant concentration, are used as the basis for locating the source of the anomaly, indicating that subsequent attention should be paid to data changes in these two dimensions. If all dimensions of a sample are marked as normal, the sample is considered normal.

[0046] The information underlying the sample abnormality determination is processed to generate abnormality marker information. Based on the previous determination, for the sample "2024-09-15 10:00:00 - Site A-001," a sample abnormality marker is generated, such as "Abnormal (temperature, pollutant concentration abnormal)." This marker clearly displays the sample's abnormality and the dimensions involved. This allows for rapid screening of abnormal samples and their associated dimensions when processing large numbers of samples, improving data analysis efficiency.

[0047] The basic information for locating the source of anomalies is processed to generate indicator location information for the source of anomalies. Taking the sample "2024-09-15 10:00:00 - Site A-001" as an example, further analysis is conducted on the two anomaly dimensions of temperature and pollutant concentration. Assuming the temperature anomaly is significantly higher than the historical average for the same period, and the pollutant concentration anomaly is due to levels of a specific pollutant exceeding the normal range, the generated indicator location information for the source of anomalies might read "Temperature: Higher than the historical average for the same period; Pollutant concentration: [Specific pollutant name] exceeds the standard." This information helps personnel quickly locate the specific indicator of the anomaly, providing a strong basis for subsequent measures (such as inspecting monitoring equipment and analyzing the cause of the pollution).

[0048] S106, processing the sample abnormal state mark and abnormal source indicator positioning information to generate abnormal information of the environmental data.

[0049] In one implementation, the sample abnormality flag information is processed, the number of abnormal samples is counted, and the ratio of abnormal samples to the total number of samples is calculated to generate an abnormal sample ratio feature. For example, in an environmental monitoring data assessment, the total number of samples was 100, and 15 samples were marked as abnormal through the previous process. The ratio of abnormal samples to the total number of samples is: \(\frac{15}{100}×100\%=15\%\). This 15% is the abnormal sample ratio feature. This ratio provides an intuitive understanding of the approximate scale of abnormal samples in this monitoring data.

[0050] The distribution of abnormal samples in different sample groups is analyzed and processed to generate abnormal sample distribution characteristics. The samples are divided into three groups: A, B, and C according to the monitoring area. Group A has 30 samples and 5 abnormal samples; Group B has 40 samples and 8 abnormal samples; Group C has 30 samples and 2 abnormal samples. By calculation, it can be obtained that the proportion of abnormal samples in Group A is , the proportion of abnormal samples in group B is =20%, the proportion of abnormal samples in group C is The proportion of abnormal samples in these different groups, as well as the specific distribution of abnormal samples in each region, constitute the abnormal sample distribution characteristics. From this, we can see that the abnormality in Group B is relatively more serious, and the data of Group B will be the focus of future attention.

[0051] When conducting a detailed analysis of potentially abnormal samples, the anomaly is determined for each feature dimension by comparing the calculated Z-Score result with the set threshold. For example, if the absolute value of a sample's temperature-related Z-Score is greater than the set threshold—for example, if the Z-Score threshold is set to 2 and the calculated Z-Score for the temperature dimension is 3—this indicates that the sample's temperature-related data deviates from the normal range of historical data, thus identifying temperature as an anomaly-source indicator. Similarly, through similar analysis of data for dimensions such as pH and dissolved oxygen, anomalies are determined in these dimensions, and these indicators are then classified as anomaly-source indicators.

[0052] All identified anomaly source indicators are combined into a set, known as the anomaly indicator category signature. In this example, the anomaly indicator category signature is represented as {temperature, pH, dissolved oxygen}. This set representation is highly general and intuitive, clearly showing which specific indicators exhibited anomalies during the data assessment. This set allows data analysts, environmental monitoring personnel, and other relevant personnel to quickly identify the key indicator areas involved in the data anomaly, eliminating the need to search and identify anomaly indicators one by one through large amounts of complex data, greatly improving information acquisition efficiency.

[0053] Clearly defining the types and characteristics of abnormal indicators provides guidance for subsequent in-depth analysis of the causes of abnormalities. For example, once temperature, pH, and dissolved oxygen are identified as abnormal indicators, researchers can focus on these areas in a targeted manner. For temperature abnormalities, researchers can further check whether the temperature sensor is functioning properly and whether there are any special heat sources affecting the monitoring environment. For pH abnormalities, researchers can analyze whether there are any chemical discharges in the surrounding area that may affect the pH of the water, or whether there are any reagent errors during the detection and analysis process. For dissolved oxygen abnormalities, researchers can consider factors such as the flow of the water and whether there are any abnormalities in the activity of aquatic organisms to quickly locate the root cause of the problem.

[0054] The types and characteristics of abnormal indicators play a crucial role in environmental management and decision-making. If abnormalities in these indicators are detected during monitoring of a specific water area, environmental protection authorities can use this information to promptly formulate appropriate remediation measures. For temperature anomalies, regulation of industrial cooling water discharge around the area may be strengthened; for pH anomalies, neutralization measures may be considered; and for dissolved oxygen anomalies, the need for oxygenation measures to improve water quality and protect the living environment of aquatic life may be assessed.

[0055] The types and characteristics of abnormal indicators are also important criteria for evaluating monitoring data quality. If abnormalities in certain specific indicators occur frequently over multiple monitoring sessions, this may indicate systemic issues with the monitoring system's monitoring of these indicators, such as aging monitoring equipment or inaccurate calibration. This should prompt relevant personnel to promptly inspect and optimize monitoring equipment and processes to improve the accuracy and reliability of monitoring data.

[0056] Count the number of times each abnormal indicator appears in all abnormal samples, calculate its frequency of occurrence, and generate the frequency characteristics of abnormal indicators. Understanding the frequency of each abnormal indicator in all abnormal samples and finding out which abnormal indicators are more common and which appear occasionally will help determine the abnormal indicators to focus on. Among the 15 abnormal samples, temperature anomaly appeared 10 times, pH anomaly appeared 6 times, and dissolved oxygen anomaly appeared 3 times. The frequency of temperature anomaly is , the frequency of abnormal pH is =40%, the frequency of abnormal dissolved oxygen is =20%. These frequency data constitute the frequency characteristics of abnormal indicators. From the frequency, we can see that temperature anomalies are relatively more common abnormal conditions.

[0057] The abnormal sample ratio, abnormal sample distribution, abnormal indicator type, and abnormal indicator frequency features are integrated to generate a comprehensive abnormality feature. The abnormal sample ratio feature (15%), abnormal sample distribution features (Group A 16.7%, Group B 20%, Group C 6.7%), abnormal indicator type features {temperature, pH, dissolved oxygen}, and abnormal indicator frequency features (temperature 66.7%, pH 40%, dissolved oxygen 20%) are integrated. A comprehensive abnormality feature can be represented using a data structure (such as a table or dictionary) that contains this information. For example, a dictionary representation might be: {"abnormal sample ratio": 15%, "abnormal sample distribution": {"Group A": 16.7%, "Group B": 20%, "Group C": 6.7%}," "abnormal indicator type": ["temperature", "pH", "dissolved oxygen"]," and "abnormal indicator frequency": {"temperature": 66.7%, "pH": 40%, "dissolved oxygen": 20%}}.

[0058] Comprehensive anomaly characteristics are analyzed and processed to generate abnormal data processing directions and abnormal indicator troubleshooting information. The comprehensive anomaly characteristics show that Group B has a higher proportion of abnormal samples than other groups, suggesting that special circumstances may exist in the monitoring area where Group B is located. A higher proportion of abnormal samples indicates a higher concentration of data anomalies in this area, making factors such as monitoring equipment failure and unique environmental interference more likely to cause data anomalies than in other areas. Therefore, prioritizing Group B for treatment allows for more efficient detection and resolution of data anomalies, avoiding unnecessary investigations in other normal areas and saving manpower and material resources.

[0059] Temperature anomalies occur most frequently among all abnormal indicators, highlighting their importance in the overall data anomaly landscape. Frequent temperature anomalies likely indicate widespread or serious issues within the temperature monitoring process, potentially stemming from faults in the temperature sensor itself or systemic errors in the data collection, transmission, and calculation processes. Therefore, prioritizing temperature indicators for investigation helps quickly identify the primary source of anomalies, promptly addressing the core issues and effectively improving data quality.

[0060] Conduct a comprehensive inspection of Group B's monitoring equipment, including both hardware and software. Regarding hardware, check whether the temperature sensor is physically damaged or covered with dirt that would affect measurement accuracy. Check whether the sensor is accurately calibrated and whether the power supply to the device is stable to avoid measurement errors due to insufficient power. Regarding software, check whether the data acquisition program has any loopholes or errors, whether it can correctly read and record temperature data, and whether the communication between the device and the data transmission terminal is normal, and whether there is any data loss or erroneous transmission. Review the temperature monitoring data, compare and analyze historical data with current data, and check whether the data change trend is consistent with common sense. For abnormal data points, trace their collection time and environmental conditions to determine whether there are any abnormal environmental factors affecting the data. At the same time, use high-precision calibration equipment to calibrate the temperature monitoring equipment to ensure the accuracy of its measurement results. During the calibration process, follow the standard calibration process and record the calibration data and results for subsequent review and analysis.

[0061] For temperature indicators, carefully examine the data collection, transmission, and calculation processes. During the data collection phase, confirm that the sensor's installation location complies with standard specifications to avoid improper installation, which may cause the measured temperature to not accurately reflect the actual ambient temperature. During the transmission process, check whether there is any interference in the data transmission line and whether the data is transmitted completely and accurately to the data processing center. During the calculation process, verify whether the data processing algorithm is correct and whether there are any temperature data anomalies caused by algorithm errors. By checking these links one by one, the accuracy and reliability of temperature data are ensured.

[0062] While temperature anomalies are a primary concern, pH and dissolved oxygen indicators should also be considered. Routine checks should be conducted on their data processing processes, including ensuring the proper functioning of the data collection equipment. For example, check whether the pH sensor's electrodes are aging, or whether the dissolved oxygen sensor's membrane needs replacement. Check for signal loss or interference during data transmission, and verify the correctness of the formulas and parameters used in data calculations. By thoroughly examining these abnormal indicators, the quality of the entire environmental monitoring data is ensured, potential issues are avoided, and a more comprehensive understanding of environmental conditions is achieved, providing an accurate basis for environmental decision-making.

[0063] "The proportion of abnormal samples is 15%." This statistic is a quantitative indicator of anomalies within the entire monitoring data set. It provides an intuitive overall perspective, allowing relevant personnel to quickly understand the proportion of abnormal data within the total number of samples. This ratio can be used to preliminarily determine the severity of the anomaly in the monitoring data. For example, if the proportion of abnormal samples in previous monitoring was generally below 5%, but this time it reached 15%, it indicates that the current data anomaly is quite prominent and requires sufficient attention. This ratio provides an important basis for resource allocation. If the proportion of abnormal samples is high, more manpower, material resources, and time will be required to thoroughly investigate the cause of the anomaly. It also helps determine the priority for handling abnormal data. When resources are limited, prioritizing datasets with a high proportion of abnormal samples can more efficiently resolve key issues and improve overall data quality.

[0064] Locating the problem area, "mainly concentrated in Group B area" clearly points out the concentration of abnormal samples. This is of key significance for troubleshooting the source of the anomaly. Different monitoring areas may be affected by different environmental factors, human activities or equipment conditions. After determining that Group B area is the abnormal concentration area, relevant personnel can focus on this area and conduct in-depth analysis of its special features. For example, Group B area may be close to an industrial pollution source, or its monitoring equipment may have specific fault hazards, which may be the reasons for the appearance of abnormal data sets. From the perspective of environmental governance and monitoring optimization, this information provides direction for the formulation of targeted measures. If it is subsequently discovered that the data anomaly is caused by emissions from a factory in Group B area, then the factory can be subject to key supervision and inspection, and required to rectify its emission behavior; if it is a problem with the monitoring equipment, the equipment in this area can be repaired, replaced or calibrated in a centralized manner, thereby effectively improving the data monitoring quality in this area.

[0065] Identify the key indicators involved in the anomaly. The "Abnormal indicators involving temperature, pH, and dissolved oxygen" section clearly lists the specific indicators in which the anomaly occurred. These indicators are crucial parameters for measuring environmental quality, and their abnormalities can have varying degrees of impact on ecosystems and human activities. For example, abnormal temperature can affect the survival and reproduction of aquatic organisms, abnormal pH can alter the chemical properties of the water, affecting the growth of aquatic organisms and the solubility of substances, and abnormal dissolved oxygen is directly related to the respiration and survival of aquatic organisms. Clarifying these abnormal indicators allows relevant personnel to conduct a targeted analysis of their impact on the environment from a professional perspective. The importance of frequency analysis is further emphasized by the fact that "temperature anomalies occur most frequently." A high frequency indicates that temperature anomalies are likely more widespread or severe, potentially due to systemic issues. This makes temperature a primary focus when troubleshooting the cause of an anomaly. It also provides a reference for developing remedial measures. For example, if a temperature anomaly is found to be caused by equipment failure at a specific monitoring point, and this failure occurs frequently, a comprehensive overhaul or replacement of the equipment at that point is necessary to resolve most of the anomaly issues. As for pH and dissolved oxygen, although their occurrence frequency is relatively low, they cannot be ignored because they may also have a significant impact on the environment and require regular inspections to determine whether there are potential problems.

[0066] Based on the previous analysis of abnormal information, the recommendation to "focus on inspecting the temperature monitoring equipment and data processing procedures in Group B" is both reasonable and necessary. When inspecting monitoring equipment, technicians can focus on both the hardware and software aspects. Regarding hardware, they should check for damage to the temperature sensor, accurate calibration, and the proper operating environment. Regarding software, they should examine the data acquisition program for vulnerabilities and any data loss or errors during data transmission. By inspecting these aspects, they can identify the potential causes of temperature anomalies and implement timely repairs and optimizations to ensure accurate temperature data.

[0067] Troubleshooting the data processing of pH and dissolved oxygen indicators is also essential. Although the frequency of anomalies is relatively low, potential issues may remain undetected. By troubleshooting data processing, including checking the calibration of data collection equipment, the stability of data transmission, and the accuracy of calculation methods, potential issues can be promptly identified and resolved, ensuring the reliability of these indicator data. This comprehensive investigation helps relevant personnel fully understand anomalies in environmental monitoring data and implement effective measures to address them, ensuring the accuracy and effectiveness of environmental monitoring work and providing reliable data support for subsequent environmental decision-making.

[0068] This application is based on a data assessment method that fuses historical local anomalies with standard scores (LOF-ZFusion). By organically combining the local anomaly factor (LOF) algorithm with the Z-score method, the model can effectively analyze the local density differences between new data and historical data and quickly identify potential anomalies. Compared with traditional anomaly detection methods, this model can not only extract more detailed anomaly information from local data features, but also use statistical methods to conduct an overall quantitative analysis of anomaly significance, ensuring the scientificity and stability of the detection results at the overall level and indicator level. The application of the LOF-ZFusion model will provide an efficient and reliable solution for environmental monitoring data quality assessment, while providing solid technical support for environmental governance decision-making.

[0069] In one embodiment, Figure 2 As shown, the present application also provides a data evaluation device based on historical local anomalies, comprising: Acquisition module 201, used to acquire historical data and target test data in environmental monitoring; Processing module 202 is used to perform data preprocessing on historical data in environmental monitoring, calculate k-neighborhood distance, reachable distance, local reachable density, local anomaly factor, and generate reference density distribution information; process the reference density distribution information, merge the test data points with the historical data one by one to calculate the LOF value, and filter out potential abnormal samples in combination with the set threshold to generate potential abnormal sample information; for potential abnormal samples, calculate the Z-Score according to the feature dimension, and generate abnormal indicator information of each dimension in combination with the set threshold; process the potential abnormal sample information and the abnormal indicator information of each dimension, mark the abnormal state of the sample and locate the abnormal source indicator, and generate the sample abnormal state mark and abnormal source indicator location information; process the sample abnormal state mark and abnormal source indicator location information to generate abnormal information of the environmental data.

[0070] Each embodiment of this application is described in a related manner, and similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, for the data evaluation method based on historical local anomalies, electronic device, electronic device, and readable storage medium embodiment, since they are basically similar to the data evaluation method based on historical local anomalies described above, the description is relatively simple, and the relevant parts can be referred to the partial description of the data evaluation method based on historical local anomalies described above.

Claims

1. A data evaluation method based on historical local anomalies, characterized in that: include: Obtain historical data and target test data from environmental monitoring; Perform data preprocessing on historical data from environmental monitoring, calculate k-neighborhood distance, reachable distance, local reachable density, local anomaly factor, and generate reference density distribution information; Process the reference density distribution information, merge the test data points with the historical data one by one to calculate the LOF value, and filter out potential abnormal samples based on the set threshold to generate potential abnormal sample information; For potential abnormal samples, calculate the Z-Score according to the feature dimension, and combine it with the set threshold to generate abnormal indicator information for each dimension; Process the potential abnormal sample information and abnormal indicator information of each dimension, mark the abnormal state of the sample and locate the abnormal source indicator, and generate the sample abnormal state mark and abnormal source indicator location information; The abnormal status mark of the sample and the positioning information of the abnormal source indicator are processed to generate abnormal information of the environmental data.

2. The method according to claim 1, wherein Perform data preprocessing on historical data from environmental monitoring, calculate k-neighborhood distance, reachable distance, local reachable density, local anomaly factor, and generate reference density distribution information, including: Clean the historical data in environmental monitoring, remove erroneous values, duplicate values ​​and invalid data, and generate cleaned historical data information; Performing standardization processing on the cleaned historical data information to generate standardized historical data information; Process the standardized historical data information to generate k-neighborhood distance information; based on the k-neighborhood distance information and the standardized historical data information, use the formula , calculate the reachable distance of data point p relative to q and generate reachable distance information, where, represents the distance between point p and point q; According to the reachable distance information and the standardized historical data information, the formula , calculate the local reachability density of the sample point p and generate local reachability density information, where, To represent the local reachability density of sample point p, p is the sample point for which the local reachability density is currently being calculated, q is the other data points in the k-neighborhood of point p, is the set of k data points closest to point p in the data set, is the distance between point p and point q; Based on the local reachability density information, using the formula , calculate the local anomaly factor of the sample point p and generate local anomaly factor information, where, is the local anomaly factor of the sample point p, which is used to evaluate whether the point p is an anomaly point. is the local reachability density of sample point q; The k-neighborhood distance information, reachable distance information, local reachable density information and local anomaly factor information are processed to generate reference density distribution information.

3. The method according to claim 1, wherein The reference density distribution information is processed, and the test data points are merged with the historical data one by one to calculate the LOF value. The potential abnormal samples are screened out by combining the set threshold and generating potential abnormal sample information, including: Performing format unification processing on historical data and test data involved in the reference density distribution information to generate data information in a unified format; Processing the data information after the format is unified to generate processed data information; Extract test data points one by one from the processed data information, add each test point to the historical data set, and generate merged data set information; Based on the merged data set information, according to the local anomaly factor calculation formula , calculate the LOF value of each test data point and generate LOF value information; The LOF value of each test point is compared with the preset threshold. If the LOF value is greater than the preset threshold, the corresponding test point is marked as a potential abnormal sample and potential abnormal sample information is generated.

4. The method according to claim 1, wherein For potential abnormal samples, the Z-Score is calculated according to the feature dimension, and combined with the set threshold, abnormal indicator information of each dimension is generated, including: Extract and filter the characteristic dimension information of potential abnormal samples to generate sample data information of each dimension feature; For the sample data information of each dimension feature, calculate the mean and standard deviation of each feature dimension in the historical data based on historical data to generate historical mean information and historical standard deviation information; Based on the sample data information, historical mean information and historical standard deviation information of each dimension feature, the Z-Score calculation formula is used , calculate the Z-Score value of each feature dimension and generate the Z-Score value information of each dimension feature, where, For the The Z-Score value of the feature dimension, For potential abnormal samples The sample data value of feature dimensions, For the The mean of feature dimensions in historical data, For the The standard deviation of each feature dimension in historical data; Compare the absolute value of the Z-Score value of each dimension feature with the Z-Score judgment threshold. If the absolute value of the Z-Score is greater than the set threshold, the feature dimension is marked as abnormal, and abnormal marking information for each dimension is generated; The abnormal marking information of each dimension is summarized and sorted to generate abnormal indicator information of each dimension.

5. The method according to claim 4, wherein Process the potential abnormal sample information and abnormal indicator information of each dimension, mark the sample abnormal status and locate the abnormal source indicator, and generate the sample abnormal status mark and abnormal source indicator location information, including: Extracting identification information of the potential abnormal sample from the potential abnormal sample information; Extract abnormal label information and abnormal dimension information of each dimension of each potential abnormal sample from the abnormal indicator information of each dimension; Process the identification information of potential abnormal samples, abnormal label information of each dimension, and abnormal dimension information. If at least one dimension of the sample is marked as abnormal, generate information based on the abnormal state of the sample; otherwise, it is judged as normal. At the same time, the dimension information marked as abnormal is used as the basic information for locating the source of the abnormality; Processing the sample abnormal state judgment basis information to generate sample abnormal state marking information; The basic information of abnormal source positioning is processed to generate abnormal source indicator positioning information.

6. The method according to claim 1, wherein Process the sample abnormal status mark and abnormal source indicator location information to generate abnormal information of environmental data, including: Process the sample abnormal status mark information, count the number of abnormal samples, calculate the ratio of the number of abnormal samples to the total number of samples, and generate the abnormal sample ratio feature; Analyze and process the distribution of abnormal samples in different sample groups to generate abnormal sample distribution characteristics; Process the abnormal source indicator location information to generate abnormal indicator type characteristics; Count the number of times each abnormal indicator appears in all abnormal samples, calculate its frequency of occurrence, and generate the abnormal indicator frequency feature; Integrate the abnormal sample proportion characteristics, abnormal sample distribution characteristics, abnormal indicator type characteristics, and abnormal indicator frequency characteristics to generate comprehensive abnormal characteristics; Analyze and process comprehensive abnormal characteristics to generate abnormal data processing directions and abnormal indicator troubleshooting information; The abnormal data processing direction and abnormal indicator investigation information are processed to generate abnormal information of environmental data.

7. A data evaluation device based on historical local anomalies, characterized in that: The device comprises: Acquisition module, used to obtain historical data and target test data in environmental monitoring; The processing module is used to perform data preprocessing on historical data in environmental monitoring, calculate the k-neighborhood distance, reachable distance, local reachable density, and local anomaly factor, and generate reference density distribution information; process the reference density distribution information, merge the test data points with the historical data one by one to calculate the LOF value, and filter out potential abnormal samples in combination with the set threshold to generate potential abnormal sample information; for potential abnormal samples, calculate the Z-Score according to the feature dimension, and generate abnormal indicator information of each dimension in combination with the set threshold; process the potential abnormal sample information and the abnormal indicator information of each dimension, mark the abnormal state of the sample and locate the abnormal source indicator, and generate the sample abnormal state mark and abnormal source indicator location information; process the sample abnormal state mark and abnormal source indicator location information to generate abnormal information of the environmental data.

8. An electronic device, characterized in that: include: a first processor; and a memory for storing executable instructions of the first processor; The first processor is configured to execute the data evaluation method based on historical local anomalies according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the second processor, the data evaluation method based on historical local anomalies according to any one of claims 1 to 6 is implemented.

Citation Information

Cited By

  • LIMS platform data cleaning method and system based on metadata

    CN121614467A

  • Metadata-based lims platform data cleaning method and system

    CN121614467B