Measurement abnormal data detection method and system based on rule engine

Abstract

The invention relates to the technical field of anomaly detection, in particular to a measurement abnormal data detection method and system based on a rule engine, and the method comprises the following steps: extracting a transmission path and a timestamp sequence based on a source point number, a transmission link number and a real-time state code of a measurement data packet, comparing a data flow direction path with a preset abnormal flow direction, and determining a data flow direction; and identifying an abnormal mode of which the risk path fitting degree exceeds a preset threshold value, and obtaining an abnormal flow direction feature set. According to the method, abnormal point locations in the data can be effectively identified and corrected by monitoring the change of the submission times and the activeness of the real-time measurement data of the power grid and the data synchronism, and the dynamic abnormal threshold setting and real-time data analysis further enhance the capability of capturing the abnormal trend, can adjust the processing strategy in time, and improve the accuracy of data processing. The method has the advantages that accumulation and propagation of error data are avoided, data redundancy and error repetition are effectively reduced due to generation of a data calibration set and application of data fingerprints, and data quality is guaranteed.

Description

Technical Field

[0001] The present invention relates to the technical field of anomaly detection, and in particular to a measurement anomaly data detection method and system based on a rule engine. Background Art

[0002] The field of anomaly detection technology includes methods and systems for identifying outliers or abnormal patterns in data sets. The core content is to analyze the normal behavior patterns of data through various algorithms and technical means, and detect abnormal data points that deviate from the pattern. Anomaly detection is widely used in many industries, such as financial fraud detection, network security, health monitoring, power grid measurement and industrial systems. In this field, the quality of data directly affects the accuracy and reliability of anomaly detection, and efficient data processing and analysis technologies are required to process and optimize large amounts of data.

[0003] Among them, the measurement abnormal data detection method of the rule engine refers to the use of predefined rules to identify abnormal data in the real-time measurement data set of the power grid. This method sets a series of rules to determine whether the data is abnormal based on specific business logic and data characteristics. The design of the rule engine includes the detection of problems such as missing data collection points, abnormal data values, and data duplication. Data verification is performed based on clear rule conditions and thresholds, and does not involve complex data processing or model adjustment. Through the rule engine, anomalies can be instantly identified and processed directly at the data input stage, improving the efficiency and accuracy of data processing.

[0004] In actual operation, existing technologies mainly rely on predefined rules and static thresholds to identify anomalies, which lack flexibility and adaptability in dynamically changing data environments. For example, in the field of financial fraud detection or network security, fixed rules cannot adapt to emerging fraud methods or attack patterns in a timely manner, resulting in detection failure or increased false alarm rates. Existing methods require a long processing time when processing large-scale data, which causes delays in application scenarios that require rapid response, affecting overall system performance and user experience. Therefore, this processing mode that relies on fixed rules faces the dual challenges of poor adaptability and low processing efficiency in the modern data-driven environment. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a measurement abnormal data detection method and system based on a rule engine.

[0006] In order to achieve the above object, the present invention adopts the following technical solution, a measurement abnormal data detection method based on a rule engine, comprising the following steps: S1: Based on the source point number, transmission link number and real-time status code of the measurement data packet, the transmission path and timestamp sequence are extracted, the data flow path is compared with the preset abnormal flow direction, the abnormal mode in which the risk path fit exceeds the preset threshold is identified, and the abnormal flow feature set is obtained; S2: Based on the abnormal flow feature set, monitor the activity status of the measurement and collection points, identify the number of data submissions and changes in activity, analyze the synchronization of adjacent point data, extract related point information, identify abnormal points, and form an abnormal point fluctuation map; S3: using the abnormal point fluctuation graph, monitoring the data flow, setting a dynamic abnormal threshold, comparing the real-time data deviation, analyzing the fluctuation amplitude, and obtaining an abnormal trend identification table; S4: using the abnormal trend identification table, extracting the real-time data of the neighboring points, compensating the continuous abnormal points, marking the measurement and collection points of the cooling state, and generating a data calibration set; S5: Through the data calibration set, combined with the ID, timestamp and key fields of the collection point, the data fingerprint is extracted, the cache fingerprint is compared, duplicate data is eliminated, the sampling frequency is adjusted, and the measurement data intelligent processing result is obtained.

[0007] As a further solution of the present invention, the abnormal flow feature set includes the flow deviation exceeding the limit path, the abnormal transmission time interval point, and the risk path matching abnormal mode; the abnormal point fluctuation diagram includes the active state fluctuation trend, the data submission frequency change trend, and the adjacent point data synchronization abnormal distribution; the abnormal trend identification table includes the data fluctuation amplitude exceeding the limit point, the change rate abnormal point, and the data distribution offset exceeding the threshold point; the data calibration set includes the repair data record, the compensation abnormal point data, and the cooling state marking point; the measurement data intelligent processing results include repeated data ratio statistics, sampling frequency adjustment strategy, and equipment self-test trigger record.

[0008] As a further solution of the present invention, the step of acquiring the abnormal flow direction feature set is specifically as follows: S111: extracting the transmission path and timestamp sequence based on the source point number, transmission link number and real-time status code of the measurement data packet, and establishing a data flow sequence; S112: using the data flow sequence, comparing the preset abnormal flow direction, calculating the flow deviation, identifying the transmission time interval of adjacent nodes, and calculating the deviation with the standard time interval, extracting the data packet number and path exceeding the time threshold range, establishing the corresponding relationship between the path deviation degree and the time deviation, and obtaining the path deviation and time deviation set; S113: calling the path deviation and time deviation set, evaluating the correlation between the path deviation degree and the time deviation, using the formula: ; Calculate the risk path fit, compare the threshold, screen abnormal modes, extract the features of abnormal modes, and obtain the abnormal flow feature set; in, is the risk path fit, is the path deviation, and is the weight parameter, is the time deviation.

[0009] As a further solution of the present invention, the step of obtaining the abnormal point fluctuation map is specifically as follows: S211: Based on the abnormal flow feature set, monitor the activity status of the measurement and collection points, count the number of data submissions of each measurement and collection point within a fixed time window, identify the submission frequency, calculate the change rate of the point submission frequency, call the set submission frequency change threshold, determine whether the point activity has a sudden change, and obtain the activity change record; S212: Analyze the synchronization of adjacent point data through the activity change record, compare the submission times of adjacent points in the same time window, calculate the time deviation value of adjacent points, and obtain the adjacent point time deviation identification result; S213: Using the adjacent point time deviation identification result, extract the associated point information, filter the point set whose time deviation value exceeds the set threshold, identify the distribution range of the measurement and collection points, and use the formula: ; Calculate the degree of aggregation of abnormal points, filter the abnormal point set, and obtain the abnormal point fluctuation map; in, Represents the degree of abnormal point aggregation, Representative The spatial coordinates of the abnormal points, Represents the mean coordinates of the abnormal points, Represents the number of abnormal points.

[0010] As a further solution of the present invention, the step of obtaining the abnormal trend identification table is specifically as follows: S311: Based on the abnormal point fluctuation diagram, multiple time series intervals of the monitoring data stream are analyzed, the fluctuation amplitude, change rate and data distribution characteristics in each interval are extracted, and the mean, standard deviation and change rate of the fluctuation amplitude are calculated, and the fluctuation characteristic coefficient is obtained by comparing the mean difference and standard deviation change trend of adjacent intervals; S312: Using the fluctuation characteristic coefficient and the set dynamic abnormal threshold, analyze the deviation of real-time data, using the formula: ; Calculate the degree of data deviation. When the set threshold is exceeded, the data point is marked as an abnormal fluctuation point, and the proportion of multiple abnormal fluctuation points in the differentiated time interval is counted to obtain a quantitative value of the degree of deviation; in, Represents real time The degree of data deviation, is the real-time data value, is the mean of the previous interval, is the standard deviation of the previous interval, is a constant, is the volatility weight factor; S313: Calculate the rate of change of abnormal fluctuation trends in multiple time periods in combination with the time distribution of abnormal fluctuation points, identify the degree of deviation of the abnormal distribution, and use the quantified value of the degree of deviation and the amplitude of change to evaluate and obtain an abnormal trend identification table.

[0011] As a further solution of the present invention, the step of obtaining the data calibration set is specifically: S411: using the abnormal trend identification table, extracting the real-time data of the neighborhood points, performing continuity detection on the real-time data, screening abnormal data points, and identifying the data fluctuation trend according to the data change amplitude and time series to obtain an abnormal data point sequence; S412: Repair the abnormal points through the abnormal data point sequence, identify the error range of the repaired data, and correct the data points whose deviations exceed the error range, using the formula: ; Calculate the data correction deviation and obtain the data repair record; in, Represents the data correction deviation, Representative The repair data value of points, Representative The repair data value of points, represents the data mean, represents the total number of data points; S413: Using the data repair record, compensating for the continuous abnormal points, calculating the data recovery rate, marking the measurement and collection points in the cooling state, and generating a data calibration set.

[0012] As a further solution of the present invention, the steps of obtaining the measurement data intelligent processing result are specifically: S511: extracting data fingerprints through the data calibration set, combining the ID, timestamp and key fields of the collection points, calculating hash values, comparing cache fingerprints, eliminating duplicate data, and obtaining a filtered data fingerprint set; S512: using the filtered data fingerprint set, calculating the ratio of the total amount of data before and after filtering, obtaining the repeated data ratio, comparing it with the set threshold, determining whether to adjust the sampling frequency, and if it exceeds the set threshold, adjusting the frequency, triggering the device self-check, updating the sampling parameters, and obtaining the adjusted sampling frequency parameters; S513: Based on the adjusted sampling frequency parameters and in combination with the equipment self-test record, the error range is identified, the error change is analyzed, the error threshold is compared, the affected measurement data is screened, and the measurement data intelligent processing result is obtained.

[0013] The measurement abnormal data detection system based on the rule engine is used to execute the above-mentioned measurement abnormal data detection method based on the rule engine, and the system includes: The data flow analysis module obtains the source point number, transmission link number and status code of the measured data packet, calculates the flow deviation, filters the data packets that exceed the flow deviation threshold, identifies the transmission time interval of adjacent nodes, calculates the time interval deviation value, and obtains the abnormal flow feature set; The abnormal point monitoring module uses the abnormal flow feature set to statistically measure the submission times and activity change rate of the collection points, compare the submission time of adjacent point data, extract the related point information, and obtain the abnormal point fluctuation map; The data comparison module uses the abnormal point fluctuation graph to set the dynamic abnormal threshold, calculate the data offset and compare the threshold, identify the fluctuation amplitude and change, and obtain the abnormal trend identification table; The deviation degree identification module uses the abnormal trend identification table to extract the real-time data of the neighborhood points, identify the deviation degree of data distribution, analyze the trend change, compensate for the continuous abnormal points, and obtain the data calibration set; The data evaluation module uses the data calibration set to compare data fingerprints and cache fingerprints, remove duplicate data, adjust the sampling frequency, trigger device self-check, and obtain intelligent processing results of measurement data.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are: In the present invention, anomaly detection is optimized by accurately tracking the flow direction of real-time measurement data of the power grid and active status monitoring, and the risk path is determined by using the deviation of the data flow direction and the transmission time interval deviation, so as to identify potential abnormal data flows in real time. This real-time monitoring mechanism significantly improves the response speed and accuracy of data processing. By monitoring the changes in the number of data submissions and activity, as well as data synchronization, it can effectively identify and correct abnormal points in the data. This dynamic abnormal threshold setting and real-time data analysis further enhances the ability to capture abnormal trends, and can adjust the processing strategy in time to avoid the accumulation and propagation of erroneous data. The generation of data calibration sets and the application of data fingerprints effectively reduce data redundancy and duplication of errors, ensure data quality, and bring higher data integrity and reliability to anomaly detection based on detailed data flow monitoring and the application of real-time processing logic. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the workflow of the present invention; Figure 2 A flowchart of obtaining an abnormal flow direction feature set in the present invention; Figure 3 A flow chart of obtaining an abnormal point fluctuation diagram in the present invention; Figure 4 A flowchart of obtaining an abnormal trend identification table in the present invention; Figure 5 A flowchart of obtaining a data calibration set in the present invention; Figure 6 The figure is a flow chart of obtaining the results of intelligent processing of measurement data in the present invention. DETAILED DESCRIPTION

[0016] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0017] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0018] Example 1: Please refer to Figure 1The present invention provides a technical solution, a measurement abnormal data detection method based on a rule engine, comprising the following steps: S1: Based on the source point number, transmission link number and real-time status code of the measurement data packet, the transmission path and timestamp sequence are extracted, the node information of the measurement data collection, transmission and processing links is recorded, the data flow path is compared with the preset abnormal flow direction, the flow deviation is calculated, the deviation between the transmission time interval of adjacent nodes and the standard time interval is identified, the abnormal mode in which the risk path fitting degree exceeds the preset threshold is identified, and the abnormal flow feature set is obtained; S2: Based on the abnormal flow feature set, monitor the active status of the measurement and collection points, identify the number of data submissions and changes in activity, analyze the synchronization of adjacent point data, extract related point information, identify abnormal points, and form an abnormal point fluctuation map; S3: Use the abnormal point fluctuation diagram to monitor the data flow, set the dynamic abnormal threshold, compare the real-time data deviation, analyze the fluctuation amplitude, change rate and data distribution deviation degree, and obtain the abnormal trend identification table; S4: Use the abnormal trend identification table to extract the real-time data of the neighboring points, compensate for the continuous abnormal points, mark the measurement and collection points of the cooling state, and generate a data calibration set; S5: Through the data calibration set, combined with the ID, timestamp and key fields of the collection point, the data fingerprint is extracted, the cache fingerprint is compared, duplicate data is eliminated, the duplicate data ratio is calculated, the sampling frequency is adjusted and the device self-check is triggered to obtain the intelligent processing results of the measurement data; The abnormal flow feature set includes the flow deviation exceeding the limit path, the abnormal transmission time interval point, and the risk path matching abnormal mode. The abnormal point fluctuation diagram includes the active state fluctuation trend, the data submission frequency change trend, and the abnormal distribution of adjacent point data synchronization. The abnormal trend identification table includes the data fluctuation amplitude exceeding the limit point, the change rate abnormal point, and the data distribution offset exceeding the threshold point. The data calibration set includes the repair data record, the compensation abnormal point data, and the cooling state marking point. The intelligent processing results of the measurement data include the repeated data ratio statistics, the sampling frequency adjustment strategy, and the equipment self-test trigger record.

[0019] See also Figure 2 ,The specific steps for obtaining the abnormal flow feature set are: S111: extracting the transmission path and timestamp sequence based on the source point number, transmission link number and real-time status code of the measurement data packet, and establishing a data flow sequence; It is necessary to extract the source point number from multiple data packets and arrange them in chronological order to construct the initial data flow sequence. During the extraction process, the source point number of each data packet is stored, and the source numbers of subsequent data packets are read in sequence and sorted in timestamp order. At the same time, the transmission link number is extracted, and the association between the source point number and the target receiving point number is identified. The node order passed by each transmission path is determined by cross-comparison to establish a complete transmission path sequence. During the construction process, for the case where multiple data packets are sent to the same target node at the same time, it is necessary to sort by timestamp, distinguish the data packets according to the arrival time, and number each path to ensure that the path sequence is correct. The real-time status code is used to identify the transmission status of the data packet, including normal transmission, loss, delay, repeated transmission, etc. When constructing the data flow sequence, the status code of the data packet needs to be associated with the transmission path for storage. For data packets in abnormal status, the specific status is additionally marked, and the path traversed is recorded, including the transmission path, transmission link and timestamp of each data packet, and it can accurately reflect the flow of data between different transmission nodes to establish a data flow sequence.

[0020] S112: using the data flow sequence, comparing the preset abnormal flow direction, calculating the flow deviation, identifying the transmission time interval of adjacent nodes, and calculating the deviation with the standard time interval, extracting the data packet number and path that exceeds the time threshold range, establishing the corresponding relationship between the path deviation degree and the time deviation, and obtaining the path deviation and time deviation set; It is necessary to set a preset abnormal flow direction, including abnormal path combinations and abnormal data packet transmission behaviors. By traversing all data packet paths in the data flow sequence, they are compared with the preset abnormal flow direction. For the matched paths, the flow deviation is calculated, and the number of node differences between the path of the data packet in the current data flow sequence and the normal path is calculated. At the same time, the degree of deviation of the number of hops in the path is considered, and the path deviation is expressed in percentage form. For the case where the path difference exceeds the set deviation threshold, the data packet path is marked as an abnormal path, and the deviation degree is recorded. The transmission time interval of adjacent nodes is analyzed, and the transmission time difference of each data packet between two adjacent nodes is calculated. The calculation method is to subtract the sending time from the receiving time, obtain the time interval value between each transmission node, and compare it with the standard time interval. The standard time interval is obtained based on the transmission data statistics, and the average value and standard deviation can be taken as the benchmark value. For the data packets that exceed the set standard time interval, the number and path information are extracted, and they are classified and stored according to the degree of exceeding the time threshold range. The corresponding relationship between the path deviation degree and the time deviation is established to form a path deviation and time deviation set.

[0021] S113: calling the path deviation and time deviation set, evaluating the correlation between the path deviation degree and the time deviation, using the formula: ; Calculate the risk path fit, compare the threshold, screen abnormal modes, extract the features of abnormal modes, and obtain the abnormal flow feature set; in, is the risk path fit, is the path deviation, and is the weight parameter, is the time deviation; Parameter meaning: Path deviation : Assume that the normal path contains 10 nodes and the node deviation of the abnormal path is 3, then calculate the path deviation: ; Time deviation : Set the standard transmission time interval mean of the normal path to 50ms, the standard deviation to 10ms, and the average time deviation of the abnormal path to 20ms, then the time deviation normalization calculation is: ; Setting weight parameters , , calculate the risk path fit: ; Abnormal determination: setting risk path fitting threshold , and compare the calculation results: ; Therefore, this path is not an abnormal path; The results show that under this setting, the path deviation and time deviation are still within the acceptable range and do not exceed the abnormal threshold. If the path deviation or time deviation increases, it will lead to Values ​​exceeding 0.5 are marked as abnormal paths.

[0022] See also Figure 3 , the specific steps for obtaining the abnormal point fluctuation map are: S211: Based on the abnormal flow feature set, monitor the activity status of the measurement and collection points, count the number of data submissions of each measurement and collection point within a fixed time window, identify the submission frequency, calculate the change rate of the point submission frequency, call the set submission frequency change threshold, determine whether the point activity has a sudden change, and obtain the activity change record; To monitor the active status of the measurement and collection points, it is necessary to obtain the data submission records of all points within a specific time window. Set one hour as the time window and count the number of data submissions of each point within the time period. For different types of monitoring points, such as temperature monitoring points, voltage monitoring points, flow monitoring points, etc., the data submission frequency is different, so it is necessary to calculate the data submission frequency of the points separately and set Indicates point The submission frequency within the time window. For multiple points in a certain area, calculate the historical average of the submission frequency of all points and variance , for the current submission frequency of each point Its historical average Compare and calculate the rate of change: ; If the rate of change Exceeding the set submission frequency change threshold , then it is judged that the activity of the point has changed suddenly. For a monitoring point of an industrial equipment, the historical submission frequency mean is 10 times / hour, the variance is 2 times / hour, and the submission frequency in a certain period of time rises to 15 times / hour. The change rate is calculated as follows: ; If the threshold is set to 0.3 and the change rate of the point is greater than the threshold, the point is judged to be abnormally active and the activity change record is obtained.

[0023] S212: Analyze the synchronization of adjacent point data through activity change records, compare the submission times of adjacent points in the same time window, calculate the time deviation values ​​of adjacent points, and obtain the adjacent point time deviation identification results; Analyze the synchronization of adjacent point data, compare the data submission of multiple adjacent points in the same area within the same time window, calculate the time deviation value of each adjacent point, and and , and the time of submitting data is recorded as , define the time deviation as: ; Count the mean time deviation of all adjacent points in the entire area and variance , call the set synchronization deviation threshold To judge, if the time deviation between a point and its adjacent points Exceeding the set threshold , then the synchronization of the point data is determined to be abnormal, and the synchronization deviation threshold is set Seconds, for points and , whose data submission time is 100 seconds and 120 seconds respectively, then the time deviation is: ; Since 20 seconds is greater than the threshold of 10 seconds, the point pair is determined to be a synchronization anomaly point, and the adjacent point time deviation identification result is obtained.

[0024] S213: Using the adjacent point time deviation identification results, extract the associated point information, filter the point set whose time deviation value exceeds the set threshold, identify the distribution range of the measurement and collection points, and use the formula: ; Calculate the degree of aggregation of abnormal points, filter the abnormal point set, and obtain the abnormal point fluctuation map; in, Represents the degree of abnormal point aggregation, Representative The spatial coordinates of the abnormal points, Represents the mean coordinates of the abnormal points, Represents the number of abnormal points; Parameter acquisition method: Spatial coordinates of abnormal points : By collecting data from each point in the monitoring area, identifying abnormal points and recording their spatial coordinates, for example, using a GPS device to measure the longitude and latitude coordinates of each abnormal point; Average coordinates : The coordinates of all abnormal points are averaged. The calculation method is as follows: ; in, is the total number of abnormal points, It is The coordinates of the anomaly points; Actual example: In a certain monitoring area, 5 abnormal points are detected, and their coordinates (expressed in two-dimensional plane coordinates) are: ; Calculate the average coordinates : ; Calculate the sum of the absolute values ​​of the distances of each point from the mean coordinate: ; ; Compute the sum of the squares of the distances of each point from the mean coordinate: ; ; Calculate the value of the denominator: ; Substitute the formula to calculate the degree of aggregation: ; The results show that the degree of aggregation is 4.25. The abnormal points have a certain degree of spatial aggregation. The larger the value, the more dispersed the points are, and the smaller the value, the more concentrated the points are. Through this value, we can further analyze the distribution characteristics of the abnormal points and provide a reference for subsequent monitoring and early warning.

[0025] See also Figure 4 , the specific steps for obtaining the abnormal trend identification table are: S311: Based on the abnormal point fluctuation diagram, multiple time series intervals of the monitoring data stream are analyzed, the fluctuation amplitude, change rate and data distribution characteristics in each interval are extracted, and the mean, standard deviation and change rate of the fluctuation amplitude are calculated. By comparing the mean difference and standard deviation change trend of adjacent intervals, the fluctuation characteristic coefficient is obtained; Each time series interval of the monitoring data stream is parsed. In actual implementation, the temperature monitoring data stream of a certain device can be analyzed. The device is set to collect temperature data every 5 seconds during continuous production, with a total of 720 data points per hour and about 17,280 data points per day. It is divided into multiple time series intervals, such as one interval per hour. The fluctuation amplitude, change rate and data distribution characteristics of each interval are extracted, and the difference between the maximum and minimum values ​​of all data points in the hour interval is calculated to obtain the fluctuation amplitude. This value can reflect the range of data changes in the time interval. The highest temperature of a certain hour is set to 75°C and the lowest temperature is set to 65°C, then the fluctuation amplitude is 10°C, and the change rate between two adjacent time points is calculated, that is: ; in, Represents the value of the current data point, represents the value of the previous data point, Indicates the sampling time interval; like , , sampling time interval ,but: ; Count the mean and standard deviation of all change rates within the interval to evaluate the stability of data fluctuations and calculate the standard deviation: ; in, is the number of data points in the interval, For a single data point, As the mean of the interval, set the mean temperature of a certain hour to 70°C, including 720 data points, calculate its standard deviation, and get: ; Compare the mean differences and standard deviation trends of adjacent intervals. Set the temperature mean of the previous interval to 68°C and the current interval mean to 70°C, then the mean change is 2°C. If the standard deviation of the previous interval is 2.8°C and the current interval standard deviation is 3.2°C, then the standard deviation change is 0.4°C. Comprehensively calculate the results to obtain the fluctuation characteristic coefficient.

[0026] S312: Use the fluctuation characteristic coefficient and the set dynamic abnormal threshold to analyze the deviation of real-time data, using the formula: ; Calculate the degree of data deviation. When the set threshold is exceeded, the data point is marked as an abnormal fluctuation point, and the proportion of multiple abnormal fluctuation points in the differentiated time interval is counted to obtain a quantitative value of the degree of deviation; in, Represents real time The degree of data deviation, is the real-time data value, is the mean of the previous interval, is the standard deviation of the previous interval, is a constant, is the volatility weight factor; Detailed explanation of the formula and the process of formula calculation and derivation: Represents the current moment The data value comes from the real-time temperature sensor. The unit is set to Celsius. The current temperature of a device is set to 76.5℃. The data is read by the temperature sensor and collected every 5 seconds. Represents the mean of the previous interval data, that is, the average temperature over the past period of time (such as the past 60 minutes). The sampling frequency for the past 60 minutes is set to once every 5 seconds, and the total number of data points is 720. The temperature data in the past 60 minutes is counted to get the average value ; Represents the standard deviation of the data in the previous interval, standard deviation =2.8; To prevent the denominator from having a zero decimal, it is generally set to To avoid numerical instability; is the fluctuation weight factor, which indicates the degree of influence of the fluctuation of the data point. This factor is set according to the fluctuation amplitude of the data. If the temperature fluctuation amplitude of the previous interval is large, a high weight value is set; if the fluctuation is small, a low weight value is set. ; Substituting the parameters into the formula: ; The results show that the deviation degree of the real-time data point is 2.035. If this value exceeds the set abnormal threshold (such as 1.8), the current data point is judged as an abnormal fluctuation point. By counting the proportion of multiple abnormal fluctuation points, the changes in abnormal trends can be further identified.

[0027] S313: Calculate the abnormal fluctuation trend change rate in multiple time periods in combination with the time distribution of the abnormal fluctuation points, identify the deviation degree of the abnormal distribution, and use the deviation degree quantification value and the change amplitude for evaluation to obtain an abnormal trend identification table; Combined with the time distribution of abnormal fluctuation points, the rate of change of abnormal fluctuation trend in each time period is calculated, and the degree of deviation of abnormal distribution is calculated. The quantitative values ​​of the degree of deviation per hour in the past 24 hours are set as follows: , , ; By calculating the rate of change in adjacent time intervals: ; in, A quantitative value for the degree of deviation in the current time period. is a quantitative value of the degree of deviation in the previous time period. is the time interval; set up Hours, calculate the rate of change in a certain period, such as the rate from the 4th hour (7.2%) to the 5th hour (8.0%) is: ; Calculate the degree of deviation of abnormal distribution, use the data of the past N periods, and calculate the mean and standard deviation , if the deviation value of a certain period Beyond , it is judged that the abnormal fluctuation deviation in this period is large, and the average value of the past 10 hours is set , standard deviation , then the abnormality judgment threshold is: ; If the quantified value of the degree of deviation in a certain hour exceeds 7.8%, the abnormal fluctuation deviation of that hour is large. The number of occurrences and time distribution of abnormal deviations in all time intervals are counted to obtain an abnormal trend identification table.

[0028] See also Figure 5 , the steps to obtain the data calibration set are as follows: S411: using an abnormal trend identification table, extracting real-time data of neighboring points, performing continuity detection on the real-time data, screening abnormal data points, and identifying data fluctuation trends according to the data change amplitude and time series to obtain an abnormal data point sequence; Extract real-time data of neighborhood points. For the data extraction process, obtain the real-time data stream of all measuring points in the target monitoring area, and classify them according to the spatial distribution of the measuring points and the data time series to ensure that all data have timestamps and measuring point numbers. Perform continuity detection on the extracted data, and use the ratio of the time interval to the data change amplitude to calculate the fluctuation rate of the data of a single measuring point. If the fluctuation rate exceeds the set threshold, the data point is judged to be abnormal and stored in the abnormal data point sequence. The set fluctuation rate threshold is calculated based on the change range of the monitoring data. If the normal data fluctuation range of the measuring points in a certain area is ±2.5℃ and the data collection time interval is 5s, the normal change rate range can be calculated. ,If the temperature change rate of a certain measuring point exceeds 0.5, it is marked as an abnormal data point, and an abnormal data point sequence is obtained.

[0029] S412: Repair the abnormal points through the abnormal data point sequence, identify the error range of the repaired data, and correct the data points whose deviations exceed the error range, using the formula: ; Calculate the data correction deviation and obtain the data repair record; in, Represents the data correction deviation, Representative The repair data value of points, Representative The repair data value of points, represents the data mean, represents the total number of data points; Formula parameter description: Represents the data correction deviation, which measures the fluctuation of the repaired data and is used to determine whether the repaired data is stable; Representative The repaired data value of the point is the data after interpolation or fitting of the abnormal point; Representative The repair data value of each point is used to calculate the change rate of adjacent points; Represents the data mean, calculates the average value of all points in the repaired data set, and represents the central trend of the data series; Parameter assignment and calculation: Set the repair data point sequence of a certain measuring point to: 32.5℃, 33.0℃, 32.8℃, 33.2℃, 32.9℃, and calculate the data mean: ; Calculate the cumulative deviation of the data change rate: ; Calculate the variance of the data: ; ; ; Calculate the data correction deviation: ; The results show that the data correction deviation is 0.8. If the set deviation reference value is 0.75℃, the current repair data still fluctuates slightly and needs further adjustment. Otherwise, the data repair can be considered completed and enter the next step of data compensation and cooling status marking.

[0030] S413: using the data repair record, compensating for the continuous abnormal points, calculating the data recovery rate, marking the measurement and collection points in the cooling state, and generating a data calibration set; Compensate for continuous abnormal points, perform continuity analysis on the repaired data points, extract time series data, calculate the data recovery rate, and define the recovery rate as the absolute value of the data change per unit time. If the recovery rate remains stable or tends to decrease in multiple continuous time steps, it indicates that the data has returned to normal. If the recovery rate fluctuates violently or there are too many continuous abnormal points, a compensation method is required to adjust the repair value. The recovery rate threshold is set based on the monitoring data and equipment operating parameters. For temperature data, the change rate range of all normal measuring points in the past hour can be counted, and the average plus two Times the standard deviation is used as the recovery threshold. If the average rate of change of normal measurement point data is 0.3℃ / s and the standard deviation is 0.15℃ / s, the recovery threshold can be set to 0.3+2×0.15=0.6. If the recovery rate of the repaired data exceeds 0.6℃ / s, it is still in an abnormal state and needs to be further adjusted and corrected. The compensation value is calculated and the trend of the data is fitted using the time series analysis method. The weighted calculation is performed based on the real-time data. If the recovery rate is lower than the set threshold, the data is judged to have recovered and stabilized, and the measurement and collection points in the cooling state are marked to generate a data calibration set.

[0031] See also Figure 6 , the specific steps for obtaining the intelligent processing results of measurement data are: S511: extract data fingerprints through the data calibration set, combine the ID, timestamp and key fields of the collection points, calculate hash values, compare cache fingerprints, remove duplicate data, and obtain a filtered data fingerprint set; Based on the ID, timestamp and key fields of the collection point, the original data fingerprint is extracted. Each data fingerprint is generated by hash calculation based on the unique identifier, timestamp and key field value of the data entry. In actual scenarios, such as fingerprint extraction of industrial sensor data, each data collection point will contain a timestamp, device ID and measurement value. If a sensor data includes ID001, time 2025-02-2010:00:00, and measurement value 45.6, the data fingerprint can be expressed as hash (ID001+2025-02-2010:00:00+45.6) After calculating the hash value, it is necessary to compare it with the fingerprint data in the cache to determine whether the data already exists in the cache fingerprint library. In the specific comparison process, the query can be accelerated through the index structure, such as using the B+ tree index to store hash values ​​to reduce the amount of calculation for traversal search. The cache fingerprint library already has fingerprint hashes A1, A2, and A3. The currently calculated hash value A2 matches the fingerprint in the cache library, then the data is determined to be duplicate data and is removed. In batch data processing, removing duplicate data can effectively reduce the burden of data storage, optimize the efficiency of data analysis and calculation, and obtain a filtered data fingerprint set.

[0032] S512: using the filtered data fingerprint set, calculating the ratio of the total amount of data before and after filtering, obtaining the duplicate data ratio, comparing it with the set threshold, determining whether to adjust the sampling frequency, and if it exceeds the set threshold, adjusting the frequency, triggering the device self-check, updating the sampling parameters, and obtaining the adjusted sampling frequency parameters; Count the number of data entries before and after deletion, and calculate the data duplication ratio. The duplication ratio can be obtained by calculating the ratio of the number of removed data items to the total number of data items. Set the original data items to 10,000, and there are 8,000 items left after removal. The duplication ratio is (10,000-8,000) / 10,000=20%. The data duplication ratio needs to be compared with the preset threshold to determine whether the current sampling frequency needs to be adjusted. If the threshold is set to 15%, the current duplication ratio exceeds the threshold and the sampling frequency needs to be reduced. In the industrial sensor data scenario, if a device collects data 100 times per minute, if the duplication ratio exceeds 15%, the sampling frequency is adjusted to 80 times / minute, and the device self-test is triggered synchronously. The self-test content includes sensor status check, data transmission stability test, etc. After adjustment, it is necessary to re-store the updated sampling parameters, such as the new sampling interval, data storage period, etc., to obtain the adjusted sampling frequency parameters.

[0033] S513: Based on the adjusted sampling frequency parameters and in combination with the equipment self-test record, the error range is identified, the error change is analyzed, the error threshold is compared, the affected measurement data is screened, and the measurement data intelligent processing result is obtained; Combined with the equipment self-test records, the measured data under different sampling frequencies are compared to calculate the measurement error range. The error calculation can analyze the data stability through statistical indicators such as the mean and variance of the data. The mean of the data measured under the original sampling frequency is set to 50.2, and the mean is 49.8 after adjusting the sampling frequency. The error is |50.2-49.8|=0.4. Further analysis of the error trend shows that if the error is stable within 0.5 in a short period of time, it is determined that the error fluctuation is small. If it exceeds 1.0, there is a sensor drift problem. After the error calculation, the error threshold is compared. For example, if the error threshold is set to 0.8, when the error exceeds 0.8, the data needs to be marked as abnormal data and the affected data range is screened. For example, data with an error greater than 0.8 within 10 minutes are considered abnormal data to obtain the intelligent processing results of the measurement data.

[0034] The measurement abnormal data detection system based on the rule engine is used to execute the above-mentioned measurement abnormal data detection method based on the rule engine, and the system includes: The data flow analysis module obtains the source point number, transmission link number and status code of the measured data packet, calculates the flow deviation, filters the data packets that exceed the flow deviation threshold, identifies the transmission time interval of adjacent nodes, calculates the time interval deviation value, and obtains the abnormal flow feature set; The abnormal point monitoring module uses the abnormal flow feature set to statistically measure the submission times and activity change rate of the collection points, compare the submission time of adjacent point data, extract the related point information, and obtain the abnormal point fluctuation map; The data comparison module uses the abnormal point fluctuation graph to set the dynamic abnormal threshold, calculate the data offset and compare the threshold, identify the fluctuation amplitude and change, and obtain the abnormal trend identification table; The deviation degree identification module uses the abnormal trend identification table to extract the real-time data of the neighborhood points, identify the deviation degree of data distribution, analyze the trend change, compensate for the continuous abnormal points, and obtain the data calibration set; The data evaluation module uses the data calibration set to compare data fingerprints and cache fingerprints, remove duplicate data, adjust the sampling frequency, trigger device self-check, and obtain intelligent processing results of measurement data.

[0035] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A measurement abnormal data detection method based on a rule engine, characterized in that: The following steps are involved: S1: Based on the source point number, transmission link number and real-time status code of the measurement data packet, the transmission path and timestamp sequence are extracted, the data flow path is compared with the preset abnormal flow direction, the abnormal mode in which the risk path fit exceeds the preset threshold is identified, and the abnormal flow feature set is obtained; S2: Based on the abnormal flow feature set, monitor the activity status of the measurement and collection points, identify the number of data submissions and changes in activity, analyze the synchronization of adjacent point data, extract related point information, identify abnormal points, and form an abnormal point fluctuation map; S3: using the abnormal point fluctuation graph, monitoring the data flow, setting a dynamic abnormal threshold, comparing the real-time data deviation, analyzing the fluctuation amplitude, and obtaining an abnormal trend identification table; S4: using the abnormal trend identification table, extracting the real-time data of the neighboring points, compensating the continuous abnormal points, marking the measurement and collection points of the cooling state, and generating a data calibration set; S5: Through the data calibration set, combined with the ID, timestamp and key fields of the collection point, the data fingerprint is extracted, the cache fingerprint is compared, duplicate data is eliminated, the sampling frequency is adjusted, and the measurement data intelligent processing result is obtained.

2. The measurement abnormal data detection method based on rule engine according to claim 1 is characterized in that: The abnormal flow feature set includes the flow deviation exceeding the limit path, the abnormal transmission time interval point, and the risk path matching abnormal mode; the abnormal point fluctuation diagram includes the active state fluctuation trend, the data submission frequency change trend, and the abnormal distribution of adjacent point data synchronization; the abnormal trend identification table includes the data fluctuation amplitude exceeding the limit point, the change rate abnormal point, and the data distribution offset exceeding the threshold point; the data calibration set includes repair data records, compensation abnormal point data, and cooling state marking points; the measurement data intelligent processing results include repeated data ratio statistics, sampling frequency adjustment strategy, and equipment self-test trigger records.

3. The measurement abnormal data detection method based on rule engine according to claim 1 is characterized in that: The steps for obtaining the abnormal flow feature set are specifically as follows: S111: extracting the transmission path and timestamp sequence based on the source point number, transmission link number and real-time status code of the measurement data packet, and establishing a data flow sequence; S112: using the data flow sequence, comparing the preset abnormal flow direction, calculating the flow deviation, identifying the transmission time interval of adjacent nodes, and calculating the deviation with the standard time interval, extracting the data packet number and path exceeding the time threshold range, establishing the corresponding relationship between the path deviation degree and the time deviation, and obtaining the path deviation and time deviation set; S113: calling the path deviation and time deviation set, evaluating the correlation between the path deviation degree and the time deviation, using the formula: ； Calculate the risk path fit, compare the threshold, screen abnormal modes, extract the features of abnormal modes, and obtain the abnormal flow feature set; in, is the risk path fit, is the path deviation, and is the weight parameter, is the time deviation.

4. The measurement abnormal data detection method based on rule engine according to claim 3 is characterized in that: The steps for obtaining the abnormal point fluctuation map are specifically as follows: S211: Based on the abnormal flow feature set, monitor the activity status of the measurement and collection points, count the number of data submissions of each measurement and collection point within a fixed time window, identify the submission frequency, calculate the change rate of the point submission frequency, call the set submission frequency change threshold, determine whether the point activity has a sudden change, and obtain the activity change record; S212: Analyze the synchronization of adjacent point data through the activity change record, compare the submission times of adjacent points in the same time window, calculate the time deviation value of adjacent points, and obtain the adjacent point time deviation identification result; S213: Using the adjacent point time deviation identification result, extract the associated point information, filter the point set whose time deviation value exceeds the set threshold, identify the distribution range of the measurement and collection points, and use the formula: ； Calculate the aggregation degree of abnormal points, filter the abnormal point set, and obtain the abnormal point fluctuation map; in, Represents the degree of abnormal point aggregation. Representative The spatial coordinates of the abnormal points, Represents the mean coordinates of the abnormal points, Represents the number of abnormal points.

5. The measurement abnormal data detection method based on rule engine according to claim 4 is characterized in that: The steps for obtaining the abnormal trend identification table are specifically as follows: S311: Based on the abnormal point fluctuation diagram, multiple time series intervals of the monitoring data stream are analyzed, the fluctuation amplitude, change rate and data distribution characteristics in each interval are extracted, and the mean, standard deviation and change rate of the fluctuation amplitude are calculated, and the fluctuation characteristic coefficient is obtained by comparing the mean difference and standard deviation change trend of adjacent intervals; S312: Using the fluctuation characteristic coefficient and the set dynamic abnormal threshold, analyze the deviation of real-time data, using the formula: ； Calculate the degree of data deviation. When the set threshold is exceeded, the data point is marked as an abnormal fluctuation point, and the proportion of multiple abnormal fluctuation points in the differentiated time interval is counted to obtain a quantitative value of the degree of deviation; in, Represents real time The degree of data deviation, is the real-time data value, is the mean of the previous interval, is the standard deviation of the previous interval, is a constant, is the volatility weight factor; S313: Calculate the rate of change of abnormal fluctuation trends in multiple time periods in combination with the time distribution of abnormal fluctuation points, identify the degree of deviation of the abnormal distribution, and use the quantified value of the degree of deviation and the amplitude of change to evaluate and obtain an abnormal trend identification table.

6. The measurement abnormal data detection method based on rule engine according to claim 5 is characterized in that: The steps for obtaining the data calibration set are specifically as follows: S411: using the abnormal trend identification table, extracting the real-time data of the neighborhood points, performing continuity detection on the real-time data, screening abnormal data points, and identifying the data fluctuation trend according to the data change amplitude and time series to obtain an abnormal data point sequence; S412: Repair the abnormal points through the abnormal data point sequence, identify the error range of the repaired data, and correct the data points whose deviations exceed the error range, using the formula: ； Calculate the data correction deviation and obtain the data repair record; in, Represents the data correction deviation, Representative The repair data value of points, Representative The repair data value of points, represents the data mean, represents the total number of data points; S413: Using the data repair record, compensating for the continuous abnormal points, calculating the data recovery rate, marking the measurement and collection points in the cooling state, and generating a data calibration set.

7. The measurement abnormal data detection method based on rule engine according to claim 6 is characterized in that: The steps for obtaining the measurement data intelligent processing result are specifically as follows: S511: extracting data fingerprints through the data calibration set, combining the ID, timestamp and key fields of the collection points, calculating hash values, comparing cache fingerprints, eliminating duplicate data, and obtaining a filtered data fingerprint set; S512: using the filtered data fingerprint set, calculating the ratio of the total amount of data before and after filtering, obtaining the repeated data ratio, comparing it with the set threshold, determining whether to adjust the sampling frequency, and if it exceeds the set threshold, adjusting the frequency, triggering the device self-check, updating the sampling parameters, and obtaining the adjusted sampling frequency parameters; S513: Based on the adjusted sampling frequency parameters and in combination with the equipment self-test record, the error range is identified, the error change is analyzed, the error threshold is compared, the affected measurement data is screened, and the measurement data intelligent processing result is obtained.

8. A measurement abnormal data detection system based on a rule engine, characterized in that: According to the rule engine-based measurement abnormal data detection method according to any one of claims 1 to 7, the system comprises: The data flow analysis module obtains the source point number, transmission link number and status code of the measured data packet, calculates the flow deviation, filters the data packets that exceed the flow deviation threshold, identifies the transmission time interval of adjacent nodes, calculates the time interval deviation value, and obtains the abnormal flow feature set; The abnormal point monitoring module uses the abnormal flow feature set to statistically measure the submission times and activity change rate of the collection points, compare the submission time of adjacent point data, extract the related point information, and obtain the abnormal point fluctuation map; The data comparison module uses the abnormal point fluctuation graph to set the dynamic abnormal threshold, calculate the data offset and compare the threshold, identify the fluctuation amplitude and change, and obtain the abnormal trend identification table; The deviation degree identification module uses the abnormal trend identification table to extract the real-time data of the neighborhood points, identify the deviation degree of data distribution, analyze the trend change, compensate for the continuous abnormal points, and obtain the data calibration set; The data evaluation module uses the data calibration set to compare data fingerprints and cache fingerprints, remove duplicate data, adjust the sampling frequency, trigger device self-check, and obtain intelligent processing results of measurement data.

Images