An adaptive detection method and system for abnormal data in the Internet of Things

By calculating the gradient difference and gradient variance of the monitored data in the Internet of Things platform, dynamically generate abnormal cutoff point thresholds, solving the detection accuracy and universality problems caused by relying on fixed thresholds in the prior art, achieving high accuracy and high application abnormal data detection.

CN118842727BActive Publication Date: 2024-12-03CHENGDU TOFLY TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411320256.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-23
Publication Date
2024-12-03
Estimated Expiration
2044-09-23

AI Technical Summary

Technical Problem

The detection method of abnormal data in the existing IoT platform relies on preset fixed thresholds, resulting in poor detection accuracy and poor universality, making it difficult to effectively identify different data when there are many types of equipment, large data scale and complex data environment. Type and distribution exception.

Method used

Adaptive detection method is adopted to calculate the gradient difference and gradient variance of the monitored data, dynamically adjust the detection standards, and generate adaptive abnormal thresholds to realize abnormal data detection without a fixed threshold, which is suitable for various data types and ranges.

Benefits of technology

It improves detection accuracy and applicability, reduces false alarms and missed reports, simplifies system configuration, is suitable for real-time processing of massive monitoring data, and is suitable for diversified Internet of Things application scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118842727B_ABST
    Figure CN118842727B_ABST
Patent Text Reader

Abstract

The invention provides an adaptive detection method and system for abnormal data of an Internet of Things. The method comprises: obtaining original monitoring data, removing invalid data and then sorting to obtain a monitoring data value sequence; successively calculating the difference between a subsequent monitoring data value and a previous monitoring data value to obtain a gradient difference sequence, calculating the proportion of the gradient difference in the sum of all gradient differences to obtain a gradient difference proportion sequence; successively calculating the gradient variance of the gradient difference and its proportion in the sum of all gradient variances to obtain a gradient variance proportion sequence; multiplying the monitoring data value sequence from the first item with the gradient variance proportion sequence item by item to obtain a starting point alignment demarcation point threshold sequence; multiplying the monitoring data value sequence from the second item with the gradient variance proportion sequence item by item to obtain an end point alignment demarcation point threshold sequence; calculating the average of the sum of the starting point and end point alignment demarcation point threshold sequences to obtain an abnormal demarcation point threshold; and obtaining an abnormal data sequence by comparing the abnormal demarcation point threshold with the monitoring data value.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of Internet of Things equipment monitoring, and in particular to an adaptive detection method and system for abnormal data of the Internet of Things. Background Art

[0002] In recent years, the rapid development of Internet of Things (IoT) technology has led to the widespread application of a large number of sensors and smart devices in various industries and fields. These devices usually collect data through IoT platforms and generate massive monitoring data. These monitoring data contain both normal data and abnormal data. Abnormal data usually indicates equipment failure or problems with monitoring indicators. If not discovered and handled in time, it may cause equipment damage or major safety hazards. Therefore, the detection and analysis of abnormal data has become an extremely important task in the IoT platform.

[0003] At present, the commonly used abnormal data detection methods in the Internet of Things platform mainly rely on preset fixed thresholds. This method requires setting different thresholds according to the characteristics of the monitoring data of different devices to adapt to large and small monitoring data. However, due to the wide variety of devices connected to the Internet of Things platform and the huge amount of data, the threshold setting process is often very cumbersome, and these fixed threshold methods are not ideal in terms of detection accuracy when dealing with complex and changing data environments. In addition, traditional methods also have the problem of poor universality, that is, the anomaly detection effects for different data types or data distributions vary greatly. Summary of the invention

[0004] In order to solve the above problems, the present invention aims to provide an anomaly detection method and system that can adaptively analyze monitoring data of an Internet of Things platform. The present invention can dynamically adjust the detection standard according to the actual situation of the monitoring data, without relying on a fixed threshold, and has higher detection accuracy and applicability, thereby better serving the safe operation and maintenance of Internet of Things devices.

[0005] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention includes:

[0006] An adaptive detection method for abnormal data in the Internet of Things, comprising the steps of:

[0007] S1. Obtain the original monitoring data of the device from the IoT platform, remove invalid data, and sort the monitoring data values ​​from small to large to obtain a monitoring data value sequence;

[0008] S2, successively calculating the difference between the next monitoring data value and the previous monitoring data value to obtain a gradient difference sequence, and respectively calculating the proportion of the gradient difference in the sum of all gradient differences to obtain a gradient difference proportion sequence;

[0009] S3, successively calculating the gradient variance of the gradient difference and its proportion in the sum of all gradient variances to obtain a gradient variance proportion sequence;

[0010] S4, multiplying the monitoring data value sequence with the gradient variance ratio sequence item by item starting from the first item to obtain a starting point alignment demarcation point threshold sequence; multiplying the monitoring data value sequence with the gradient variance ratio sequence item by item starting from the second item to obtain an end point alignment demarcation point threshold sequence;

[0011] S5, calculating the average of the sum of the start point alignment demarcation point threshold sequence and the sum of the end point alignment demarcation point threshold sequence to obtain an abnormal demarcation point threshold;

[0012] S6. Obtain an abnormal data sequence by comparing the abnormal demarcation point threshold with the monitoring data value.

[0013] Preferably, the invalid data includes empty data, repeated data, out-of-bounds data and data within a specified time period.

[0014] Preferably, before the step S3 calculating the gradient variance, the step S3 also includes the following steps:

[0015] S301, setting data gradient ratio;

[0016] S302, calculating the proportion of the maximum value in the gradient difference sequence in the sum of all gradient differences to obtain the maximum data gradient proportion;

[0017] S303: If the data gradient ratio is less than the maximum data gradient ratio, continue with the subsequent steps; otherwise, the process is interrupted.

[0018] Preferably, step S6 comprises:

[0019] S601, taking the monitoring data values ​​greater than or equal to the abnormal demarcation point threshold as a first grouping sequence;

[0020] S602, taking the monitoring data values ​​less than the abnormal demarcation point threshold as a second grouping sequence;

[0021] S603: Take the grouping sequence with a smaller number of elements as an abnormal data sequence.

[0022] Preferably, step S6 comprises:

[0023] S611, drawing a two-dimensional line graph of the monitoring data value sequence according to the time flow;

[0024] S612, drawing a straight line where the abnormal demarcation point threshold is located;

[0025] S613: Compare the number of data points on both sides of the straight line, and take the data points on the side with fewer data points as abnormal data.

[0026] The present invention also provides an adaptive detection system for abnormal data of the Internet of Things, comprising:

[0027] A data acquisition module is used to obtain the original monitoring data of the device from the Internet of Things platform, sort the monitoring data from small to large values ​​after removing invalid data, and obtain a monitoring data value sequence;

[0028] A gradient difference calculation module, connected to the data acquisition module, is used to successively calculate the difference between the next monitoring data value and the previous monitoring data value to obtain a gradient difference sequence, and calculate the proportion of the gradient difference in the sum of all gradient differences to obtain a gradient difference proportion sequence;

[0029] A gradient variance calculation module, connected to the gradient difference calculation module, is used to successively calculate the gradient variance of the gradient difference and its proportion in the sum of all gradient variances to obtain a gradient variance proportion sequence;

[0030] A demarcation point threshold calculation module is connected to the data acquisition module and the gradient variance calculation module respectively, and is used to multiply the monitoring data value sequence with the gradient variance proportion sequence item by item starting from the first item to obtain a starting point alignment demarcation point threshold sequence; and multiply the monitoring data value sequence with the gradient variance proportion sequence item by item starting from the second item to obtain an end point alignment demarcation point threshold sequence;

[0031] An abnormal demarcation point determination module is connected to the demarcation point threshold calculation module and is used to calculate the average of the sum of the start point alignment demarcation point threshold sequence and the sum of the end point alignment demarcation point threshold sequence to obtain the abnormal demarcation point threshold;

[0032] The abnormal data detection module is connected to the abnormal demarcation point determination module and is used to obtain an abnormal data sequence by comparing the abnormal demarcation point threshold with the monitoring data value.

[0033] Preferably, the invalid data cleared by the data acquisition module includes empty data, duplicate data, out-of-bounds data and data within a specified time period.

[0034] Preferably, it also includes:

[0035] The abnormal preliminary judgment module has a built-in data gradient ratio, which is connected to the gradient difference calculation module and the gradient variance calculation module respectively, and is used to calculate the proportion of the maximum value in the gradient difference sequence in the sum of all gradient differences to obtain the maximum data gradient ratio, and judge whether the data gradient ratio is less than the maximum data gradient ratio. If so, the data transmission to the gradient variance calculation module is maintained; if not, the data transmission to the gradient variance calculation module is interrupted.

[0036] Preferably, the abnormal data detection module includes:

[0037] A data grouping unit, configured to use the monitoring data values ​​greater than or equal to the abnormal demarcation point threshold as a first grouping sequence, and use the monitoring data values ​​less than the abnormal demarcation point threshold as a second grouping sequence;

[0038] The first abnormal data extraction unit is connected to the data grouping unit and is used to extract a grouping sequence with a smaller number of elements as an abnormal data sequence.

[0039] Preferably, the abnormal data detection module includes:

[0040] A graph drawing unit, used for drawing a two-dimensional line graph of the monitoring data value sequence according to the time flow, and drawing a straight line where the abnormal demarcation point threshold is located;

[0041] The second abnormal data extraction unit is connected to the diagram drawing unit and is used to compare the number of data points on both sides of the straight line and extract the data points on the side with fewer data points as abnormal data.

[0042] Beneficial effects: 1. Adaptive data detection: The present invention can adaptively perform abnormal data detection according to the actual situation of the monitoring data in the Internet of Things platform, without relying on a preset fixed threshold. By dynamically calculating the data gradient difference and gradient variance, an adaptive abnormal demarcation point threshold is automatically generated, thereby realizing universal anomaly detection for different data ranges and data types.

[0043] 2. Improve detection accuracy: By performing a detailed analysis of the gradient difference and gradient variance of the monitoring data, the present invention can accurately identify abnormal data. Especially when the data distribution is complex and diverse, the detection method of the present invention has higher accuracy and reliability than the traditional threshold method, and can effectively reduce the occurrence of false positives and false negatives.

[0044] 3. Enhanced universality and versatility: The abnormal data analysis method of the present invention is applicable to various types of monitoring data, whether large or small, positive or negative, and can be effectively processed by a unified algorithm. This enables the system to be widely used in different IoT platforms and monitoring scenarios, with strong universality and versatility.

[0045] 4. Simplify system configuration: Traditional methods usually require setting thresholds for different devices or data types separately, which is complex and error-prone. The adaptive analysis system of the present invention greatly simplifies the system configuration process by automatically calculating the abnormal demarcation point threshold, reduces the risk of human intervention and setting errors, and improves the ease of use of the system.

[0046] 5. Real-time data processing: The present invention can quickly process the massive monitoring data generated on the IoT platform, and detect and filter out abnormal data in real time. This has important application value for IoT application scenarios that require immediate response, such as industrial monitoring, intelligent manufacturing, environmental monitoring, etc., and helps to provide early warning and prevent the occurrence of potential failures or disasters. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic flow chart of an adaptive detection method for abnormal data in the Internet of Things provided in a preferred embodiment of the present invention;

[0048] Figure 2 A schematic flow chart of an adaptive detection method for abnormal data in the Internet of Things provided in another preferred embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the results of obtaining raw monitoring data provided in another preferred embodiment of the present invention;

[0050] Figure 4 A schematic diagram of the result after the original monitoring data provided in another preferred embodiment of the present invention has been cleaned;

[0051] Figure 5 A schematic diagram of the results of reordering the monitoring data provided in another preferred embodiment of the present invention;

[0052] Figure 6 A schematic diagram of the results of calculating the gradient difference and the gradient difference ratio provided in another preferred embodiment of the present invention;

[0053] Figure 7 A schematic diagram of the results of calculating the gradient variance and the gradient variance ratio provided in another preferred embodiment of the present invention;

[0054] Figure 8 A schematic diagram of the result of obtaining a start point alignment demarcation point threshold sequence and an end point alignment demarcation point threshold sequence provided in another preferred embodiment of the present invention;

[0055] Fig. 9 A schematic diagram of a result of obtaining abnormal data in a grouping manner provided in another preferred embodiment of the present invention;

[0056] Fig.10 A schematic diagram of a result of obtaining abnormal data by drawing a line graph provided in another preferred embodiment of the present invention;

[0057] Fig.11 A schematic diagram of the structure of an adaptive detection system for abnormal data in the Internet of Things provided in another preferred embodiment of the present invention; DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. In the description of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings, which are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as limiting the present invention.

[0059] Embodiment 1: Figure 1 As shown, this embodiment provides an adaptive detection method for abnormal data in the Internet of Things, including the steps of:

[0060] S1. Obtain the original monitoring data of the device from the IoT platform, remove invalid data, and sort the data from small to large to obtain a monitoring data value sequence.

[0061] The original monitoring data must be independent and continuous data collected by the equipment, and there is no data with a cumulative relationship between the previous and the next. For example, the cumulative flow data of the flow meter (meter) is cumulative data, and such data is not applicable to the present invention. However, water quality indicators such as water turbidity, pH value, chemical oxygen demand, residual chlorine, conductivity, pressure, water level, flow rate and other indicators in the pipe network, as well as other temperature, concentration, air pressure, humidity, voltage, power, etc. are very suitable for the abnormal analysis of the present invention. According to actual needs, obtain the equipment monitoring data of the specified time period (nearly 3 months, nearly one month, nearly 15 days, nearly 7 days, nearly 3 days, nearly 24 hours, etc.).

[0062] Considering that the original monitoring data contains invalid data, it will reduce the quality of the data and affect the accuracy and reliability of subsequent analysis, and it is necessary to remove the invalid data in the original monitoring data. In some preferred embodiments, the invalid data includes but is not limited to empty data, repeated data, over-bounded data and specified time period data. Wherein, the empty data, that is, no data, such data cannot participate in numerical analysis, which is generally caused by acquisition failure or data access failure, so it needs to be cleaned up. The repeated data will affect the correctness of the analysis results, so it also needs to be cleaned up. The over-bounded data refers to data that obviously does not meet the expected value of the data, such as for non-negative monitoring data, it is necessary to clean up the negative data, such as pH value. The specified time period data refers to the data that needs to be proposed to be removed within the control time period according to the setting of the technician or other reasons, so as not to affect the calculation results.

[0063] S2. Successively calculate the difference between the next monitoring data value and the previous monitoring data value to obtain a gradient difference sequence, and respectively calculate the proportion of the gradient difference in the sum of all gradient differences to obtain a gradient difference proportion sequence.

[0064] The calculation formula of the gradient difference G is as follows: ; Where D is the monitoring data value sequence, and i is the position number of the element in the sequence to which the parameter belongs. It should be understood that if the number of monitoring data values ​​is n, the gradient difference is the (i+1)th monitoring data value minus the (i)th item, so the number of data of the gradient difference is n-1.

[0065] The calculation formula of the sum of the gradient differences T is as follows: ; where n is the number of elements in the sequence to which the parameter belongs.

[0066] The calculation formula of the gradient difference ratio J is as follows: .

[0067] S3. Calculate the gradient variance of the gradient difference and its proportion in the sum of all gradient variances one by one to obtain a gradient variance proportion sequence.

[0068] The calculation formula of the gradient variance U is as follows: .

[0069] The calculation formula of the sum of the gradient variance S is as follows: .

[0070] The calculation formula of the gradient variance ratio P is as follows: .

[0071] S4, multiplying the monitoring data value sequence with the gradient variance ratio sequence item by item starting from the first item to obtain a start point alignment demarcation point threshold sequence. Multiplying the monitoring data value sequence with the gradient variance ratio sequence item by item starting from the second item to obtain an end point alignment demarcation point threshold sequence.

[0072] As mentioned above, if the number of monitoring data values ​​is n, the number of gradient difference proportion data and gradient variance proportion data is n-1, which will cause the monitoring data value sequence to be unable to align with the gradient difference proportion sequence and the gradient variance proportion sequence one by one, so it is also necessary to perform specific operations on the three. Those skilled in the art can know that two alignment methods can be used in this embodiment, one is starting point alignment, and the other is end point alignment. If only one data alignment method is used for calculation, then there will inevitably be a maximum or minimum data that cannot participate in the calculation. Therefore, this embodiment considers using the following step S5 to achieve alignment.

[0073] The starting point is aligned with the demarcation point threshold L start The calculation formula is as follows: .

[0074] The endpoint alignment demarcation point threshold L end The calculation formula is as follows: .

[0075] S5. Calculate the average of the sum of the start point alignment demarcation point threshold sequence and the sum of the end point alignment demarcation point threshold sequence to obtain an abnormal demarcation point threshold.

[0076] The calculation formula of the abnormal demarcation point threshold L is as follows: .

[0077] S6. Obtaining an abnormal data sequence by comparing the abnormal demarcation point threshold with the monitoring data value. Those skilled in the art can know that after obtaining the abnormal demarcation point threshold, the relationship between the abnormal data and the abnormal demarcation point threshold can be preset according to actual conditions, thereby completing the monitoring of abnormal data.

[0078] In some preferred embodiments, the data are grouped according to the threshold value of less than or not less than the abnormal demarcation point, the number of monitoring data in each group is counted, and according to the principle that abnormal data is usually a small amount of data, the group of data with a relatively small number of data in the two groups is judged as abnormal data. Specifically, it includes:

[0079] S601. Take the monitoring data values ​​that are greater than or equal to the abnormal demarcation point threshold as a first grouping sequence.

[0080] S602: Take the monitoring data values ​​that are smaller than the abnormal demarcation point threshold as a second grouping sequence.

[0081] S603: Take the grouping sequence with a smaller number of elements as an abnormal data sequence.

[0082] In some other preferred embodiments, according to the above criterion that abnormal data is usually a small amount of data, the abnormal data can also be extracted and displayed in an intuitive way, specifically including:

[0083] S611. Draw a two-dimensional line graph of the monitoring data value sequence according to the time flow.

[0084] S612: Draw a straight line where the abnormal demarcation point threshold is located.

[0085] S613: Compare the number of data points on both sides of the straight line, and take the data points on the side with fewer data points as abnormal data.

[0086] Furthermore, different legends or colors may be used to draw normal data and abnormal data nodes to make the graph more eye-catching.

[0087] In order to further illustrate the technical idea of ​​this application, the above step-by-step calculation formula is now sorted out to obtain the following summary formula:

[0088] ;

[0089] in, The gradient variance U represents the amplitude of data change. By squaring the gradient difference of all monitoring data, the difference is amplified, and the position and amplitude of the data difference are highlighted, which facilitates better positioning of abnormal data.

[0090] It is the ratio of the gradient variance to the sum of the gradient variances. The original numerical amplitude is converted into a percentage amplitude, that is, normalization is performed to reflect the abnormal contribution ratio of the gradient difference at each data position, which is convenient for the subsequent calculation of the threshold of the abnormal data point.

[0091] and are the starting point alignment threshold L start Align the demarcation point threshold L with the end point end , since the gradient data is the difference between two adjacent data, the number of gradient data is always one less than the number of original data, and both the start point alignment and the end point alignment need to be considered. Considering that the gradient difference G represents the difference between the next data and the previous data in the sequence, when the difference between the two is large, it is more likely that at least one of the two is abnormal, but at this time it is impossible to accurately determine which one of the two is abnormal. Therefore, L is designed start and L end To solve this problem. start The latter data in L end is taken as the previous data, at this time L start and L end The value of represents the degree of abnormality when a certain data is used as the previous data and the next data (that is, the subtrahend and the minuend).

[0092] Finally, solve the point alignment demarcation point threshold L start Align the demarcation point threshold L with the end point end The idea of ​​this averaging process is to balance the calculation results under the two alignment methods to ensure the stability and reliability of anomaly detection.

[0093] The design idea of ​​the calculation method of the present invention is: first, by calculating the gradient difference of the monitoring data, the severity of the data change is captured, and the gradient difference ratio is used for normalization processing, so that the detection process can be more sensitive and more adaptable. Finally, through alignment processing and balance calculation, a demarcation point threshold that can effectively detect anomalies is obtained, thereby accurately locating anomalies in the monitoring data. The advantages of this method are its adaptability, versatility and accuracy. It can adapt to various types of data, while ensuring that it still has a high detection effect when the data distribution is complex.

[0094] Embodiment 2: Figure 2 As shown, this embodiment is developed on the basis of the above-mentioned embodiment 1.

[0095] The first mode is the "relative anomaly mode". In this mode, the system assumes that there may be anomalies in the monitoring data, so the entire detection process must be executed in full. This mode assumes that all data has the possibility of potential anomalies, so regardless of the preliminary characteristics of the data, the system will perform comprehensive anomaly detection according to predetermined algorithms and steps. Example 1 shows the complete process of this mode in detail, by gradually calculating the gradient difference, gradient variance and corresponding demarcation point threshold of the data, and finally determining and outputting the abnormal data sequence. This mode is suitable for scenarios that are highly sensitive to data and need to ensure that no potential anomalies are missed.

[0096] The second mode is the "absolute anomaly mode". In this mode, the system first makes a preliminary anomaly judgment on the monitoring data before executing the complete detection process. This preliminary judgment is intended to quickly evaluate the overall situation of the data and determine whether it may contain abnormal data. If the preliminary judgment result shows that there is an anomaly, the system will continue to perform subsequent detection steps and further analyze the data to accurately locate the anomaly point; conversely, if the preliminary judgment result shows that the data is not abnormal, the system can choose to stop subsequent detection steps according to actual needs, thereby saving computing resources and processing time. In addition, users can also choose to perform subsequent steps even if the preliminary judgment is that there is no abnormality, in order to ensure the comprehensiveness of the detection and find possible minor abnormal data. This mode is particularly suitable for scenarios with large amounts of data and limited computing resources, or for applications with high real-time requirements.

[0097] It should be noted that the "relative anomaly mode" and "absolute anomaly mode" are two parallel options. Users can flexibly select the appropriate mode for anomaly detection based on specific application scenarios, data characteristics, and system resource conditions. The two modes have their own advantages, providing flexibility and adaptability in different application scenarios, so that the anomaly detection system of the present invention can better meet the diverse IoT application needs.

[0098] This embodiment aims to provide an implementation example of an "absolute abnormal mode", which specifically includes:

[0099] Before the step S3 calculates the gradient variance, the following steps are also included:

[0100] S301, set the data gradient ratio. The data gradient ratio can be set by statistically analyzing the relevant characteristics of historical abnormal data to ensure that the set value can accurately reflect the abnormal characteristics of the data. Below, a specific example of a data gradient ratio setting implementation method is provided:

[0101] 1. Historical data collection: Collect a large amount of historical monitoring data, including known abnormal data and normal data. Ensure that the collected data is representative and can cover different devices, different time periods, and various possible abnormal situations.

[0102] 2. Statistical analysis: By performing distribution statistics on historical abnormal data, the change range and distribution pattern of abnormal data under abnormal circumstances can be determined. This can be achieved through methods such as histogram analysis and distribution fitting.

[0103] 3. Ratio threshold setting: According to the above statistical analysis results, determine a reasonable gradient ratio threshold. This threshold is usually based on the statistical characteristics of abnormal data changes. The abnormal threshold can be defined by setting a certain quantile (such as the 20% quantile) to ensure that changes within the threshold range are identified as abnormal.

[0104] 4. Verify the set results: Apply the set gradient ratio threshold to a new data set for test verification. Verify the effectiveness of the threshold by comparing the detected abnormal data with the actual situation, and make necessary adjustments based on the test results.

[0105] S302: Calculate the proportion of the maximum value in the gradient difference sequence in the sum of all gradient differences to obtain the maximum data gradient proportion.

[0106] S303, if the data gradient ratio is less than the maximum data gradient ratio, then continue with the subsequent steps, otherwise the process is interrupted. Taking the aforementioned data gradient ratio as 20 as an example, when the maximum data gradient ratio is greater than or equal to 20, then continue with the subsequent step S4 to find specific abnormal data. When the maximum data gradient ratio is less than 20, the subsequent process is interrupted.

[0107] Embodiment 3: This embodiment is developed on the basis of the above-mentioned embodiment 1. This embodiment further describes the implementation process of the present invention in conjunction with specific data examples.

[0108] 1. Create a data table in the database according to the table below to facilitate the storage and analysis of monitoring data.

[0109]

[0110] Floating point numbers are data types with decimals, which can be Number, Float, Double, Decimal, Money, etc. The corresponding type is selected according to different database software. Considering the different sizes of monitored data values, some data are decimals less than 1, so considering the compatibility and applicability of the data, there is no limit on the number of decimal places for floating point field types.

[0111] 2. Obtain the original monitoring data of the device from the IoT platform. You can specify the device monitoring data for a specified time period (recent 3 months, recent month, recent 15 days, recent 7 days, recent 3 days, recent 24 hours, etc.) and save it in the above data table. The results are as follows: Figure 3 shown.

[0112] 3. Clear invalid data. The data in row 5 is empty, the data in row 12 is 0, and the data in row 22 is completely repeated with the data in row 21. Combined with the data business rule analysis, these three types of data may be invalid data and need to be cleared. The cleaning results are as follows: Figure 4 shown.

[0113] 4. Sort the monitored data values ​​from small to large to obtain the monitored data value sequence. The result is as follows: Figure 5 shown.

[0114] 5. Calculate the gradient difference and the gradient difference ratio, where the sum of all gradient differences is 59. The result is as follows Figure 6 shown.

[0115] 6. Calculate the gradient variance and gradient variance ratio, where the sum of all gradient variances is 387. The results are as follows: Figure 7 shown.

[0116] 7. Multiply the monitoring data value sequence from the first item with the gradient variance ratio sequence item by item to obtain the starting point alignment demarcation point threshold sequence; multiply the monitoring data value sequence from the second item with the gradient variance ratio sequence item by item to obtain the end point alignment demarcation point threshold sequence; the result is as follows: Figure 8 shown.

[0117] 8. The sum of the start point alignment demarcation point threshold sequence and the sum of the end point alignment demarcation point threshold sequence are calculated to be 138.941 and 149.481 respectively. The average of the two, i.e., the abnormal demarcation point threshold, is 144.211.

[0118] 9. Next, different methods can be used to extract abnormal data. This embodiment provides examples for each method.

[0119] 9.1. After obtaining the abnormal demarcation point threshold of 144.211, traverse all the monitoring data values, determine the size relationship between the two, and divide the monitoring data into two groups. Grouping mark 1 represents the monitoring data that is less than the abnormal demarcation point threshold, and grouping mark 2 represents the monitoring data that is greater than the abnormal demarcation point threshold. The results are as follows: Fig. 9 At this time, according to the criterion that abnormal data is usually a small amount of data, it can be determined that group 1 is abnormal data. In other preferred embodiments, this step can also be implemented in a database statistical manner, and the present invention does not further limit the details of the implementation method.

[0120] 9.2. Draw a two-dimensional line graph of the monitoring data value sequence according to the time flow, and draw the straight line where the abnormal demarcation point threshold 144.211 is located as the demarcation line. The result is as follows: Fig.10 As shown. Obviously, the number of data points below the dividing line is smaller at this time. According to the principle that abnormal data is usually a small amount of data, it can be judged that it is abnormal data. It should be understood that different legends or colors can be used to draw normal data and abnormal data nodes to make the graph more eye-catching. Fig.10 The round nodes represent normal monitoring data, and the square nodes represent abnormal monitoring data.

[0121] Embodiment 4: Fig.11 As shown, this embodiment is developed on the basis of the above-mentioned embodiments 1 and 2. This embodiment provides an adaptive detection system for abnormal data of the Internet of Things, including:

[0122] A data acquisition module is used to obtain the original monitoring data of the device from the Internet of Things platform, sort the monitoring data from small to large values ​​after removing invalid data, and obtain a monitoring data value sequence;

[0123] A gradient difference calculation module, connected to the data acquisition module, is used to successively calculate the difference between the next monitoring data value and the previous monitoring data value to obtain a gradient difference sequence, and calculate the proportion of the gradient difference in the sum of all gradient differences to obtain a gradient difference proportion sequence;

[0124] A gradient variance calculation module, connected to the gradient difference calculation module, is used to successively calculate the gradient variance of the gradient difference and its proportion in the sum of all gradient variances to obtain a gradient variance proportion sequence;

[0125] A demarcation point threshold calculation module is connected to the data acquisition module and the gradient variance calculation module respectively, and is used to multiply the monitoring data value sequence with the gradient variance proportion sequence item by item starting from the first item to obtain a starting point alignment demarcation point threshold sequence; and multiply the monitoring data value sequence with the gradient variance proportion sequence item by item starting from the second item to obtain an end point alignment demarcation point threshold sequence;

[0126] An abnormal demarcation point determination module is connected to the demarcation point threshold calculation module and is used to calculate the average of the sum of the start point alignment demarcation point threshold sequence and the sum of the end point alignment demarcation point threshold sequence to obtain the abnormal demarcation point threshold;

[0127] The abnormal data detection module is connected to the abnormal demarcation point determination module and is used to obtain an abnormal data sequence by comparing the abnormal demarcation point threshold with the monitoring data value.

[0128] In some preferred embodiments, the invalid data cleared by the data acquisition module includes empty data, duplicate data, out-of-bounds data, and data within a specified time period.

[0129] In some preferred embodiments, the adaptive detection system for abnormal data of the Internet of Things further includes:

[0130] The abnormal preliminary judgment module has a built-in data gradient ratio, which is connected to the gradient difference calculation module and the gradient variance calculation module respectively, and is used to calculate the proportion of the maximum value in the gradient difference sequence in the sum of all gradient differences to obtain the maximum data gradient ratio, and judge whether the data gradient ratio is less than the maximum data gradient ratio. If so, the data transmission to the gradient variance calculation module is maintained; if not, the data transmission to the gradient variance calculation module is interrupted.

[0131] In some preferred embodiments, the abnormal data detection module includes:

[0132] A data grouping unit, configured to use the monitoring data values ​​greater than or equal to the abnormal demarcation point threshold as a first grouping sequence, and use the monitoring data values ​​less than the abnormal demarcation point threshold as a second grouping sequence;

[0133] The first abnormal data extraction unit is connected to the data grouping unit and is used to extract a grouping sequence with a smaller number of elements as an abnormal data sequence.

[0134] In some preferred embodiments, the abnormal data detection module includes:

[0135] A graph drawing unit, used for drawing a two-dimensional line graph of the monitoring data value sequence according to the time flow, and drawing a straight line where the abnormal demarcation point threshold is located;

[0136] The second abnormal data extraction unit is connected to the diagram drawing unit and is used to compare the number of data points on both sides of the straight line and extract the data points on the side with fewer data points as abnormal data.

[0137] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention to be protected. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. An adaptive detection method for abnormal data in the Internet of Things, characterized in that: Includes steps: S1. Obtain the original monitoring data of the device from the IoT platform, remove invalid data, and sort the monitoring data values ​​from small to large to obtain a monitoring data value sequence; S2, successively calculating the difference between the next monitoring data value and the previous monitoring data value to obtain a gradient difference sequence, and respectively calculating the proportion of the gradient difference in the sum of all gradient differences to obtain a gradient difference proportion sequence; S3, successively calculating the gradient variance of the gradient difference and its proportion in the sum of all gradient variances to obtain a gradient variance proportion sequence; S4, multiplying the monitoring data value sequence with the gradient variance ratio sequence item by item starting from the first item to obtain a starting point alignment demarcation point threshold sequence; multiplying the monitoring data value sequence with the gradient variance ratio sequence item by item starting from the second item to obtain an end point alignment demarcation point threshold sequence; S5, calculating the average of the sum of the start point alignment demarcation point threshold sequence and the sum of the end point alignment demarcation point threshold sequence to obtain an abnormal demarcation point threshold; S6, obtaining an abnormal data sequence by comparing the abnormal demarcation point threshold with the monitoring data value; The step S6 comprises: S601, taking the monitoring data values ​​greater than or equal to the abnormal demarcation point threshold as a first grouping sequence; S602, taking the monitoring data values ​​less than the abnormal demarcation point threshold as a second grouping sequence; S603: Take the grouping sequence with a smaller number of elements as an abnormal data sequence.

2. The method for adaptively detecting abnormal data of the Internet of Things according to claim 1, characterized in that: The invalid data includes empty data, repeated data, out-of-bounds data and data in a specified time period.

3. The adaptive detection method for abnormal data of the Internet of Things according to claim 1, characterized in that: Before the step S3 calculates the gradient variance, the following steps are also included: S301, setting data gradient ratio; S302, calculating the proportion of the maximum value in the gradient difference sequence in the sum of all gradient differences to obtain the maximum data gradient proportion; S303: If the data gradient ratio is less than the maximum data gradient ratio, continue with the subsequent steps; otherwise, the process is interrupted.

4. The method for adaptively detecting abnormal data of the Internet of Things according to claim 1, characterized in that: When step S601-S603 are not included, step S6 further includes: S611, drawing a two-dimensional line graph of the monitoring data value sequence according to the time flow; S612, drawing a straight line where the abnormal demarcation point threshold is located; S613: Compare the number of data points on both sides of the straight line, and take the data points on the side with fewer data points as abnormal data.

5. An adaptive detection system for abnormal data in the Internet of Things, characterized in that: include: A data acquisition module is used to obtain the original monitoring data of the device from the Internet of Things platform, sort the monitoring data from small to large values ​​after removing invalid data, and obtain a monitoring data value sequence; A gradient difference calculation module, connected to the data acquisition module, is used to successively calculate the difference between the next monitoring data value and the previous monitoring data value to obtain a gradient difference sequence, and calculate the proportion of the gradient difference in the sum of all gradient differences to obtain a gradient difference proportion sequence; A gradient variance calculation module, connected to the gradient difference calculation module, is used to successively calculate the gradient variance of the gradient difference and its proportion in the sum of all gradient variances to obtain a gradient variance proportion sequence; A demarcation point threshold calculation module is connected to the data acquisition module and the gradient variance calculation module respectively, and is used to multiply the monitoring data value sequence with the gradient variance proportion sequence item by item starting from the first item to obtain a starting point alignment demarcation point threshold sequence; and multiply the monitoring data value sequence with the gradient variance proportion sequence item by item starting from the second item to obtain an end point alignment demarcation point threshold sequence; An abnormal demarcation point determination module is connected to the demarcation point threshold calculation module and is used to calculate the average of the sum of the start point alignment demarcation point threshold sequence and the sum of the end point alignment demarcation point threshold sequence to obtain the abnormal demarcation point threshold; An abnormal data detection module, connected to the abnormal demarcation point determination module, is used to obtain an abnormal data sequence by comparing the abnormal demarcation point threshold with the monitoring data value; The abnormal data detection module includes: A data grouping unit, configured to use the monitoring data values ​​greater than or equal to the abnormal demarcation point threshold as a first grouping sequence, and use the monitoring data values ​​less than the abnormal demarcation point threshold as a second grouping sequence; The first abnormal data extraction unit is connected to the data grouping unit and is used to extract a grouping sequence with a smaller number of elements as an abnormal data sequence.

6. The adaptive detection system for abnormal data of the Internet of Things according to claim 5, characterized in that: The invalid data cleared by the data acquisition module includes empty data, duplicate data, out-of-bounds data and data in a specified time period.

7. The adaptive detection system for abnormal data of the Internet of Things according to claim 5, characterized in that: Also includes: The abnormal preliminary judgment module has a built-in data gradient ratio, which is connected to the gradient difference calculation module and the gradient variance calculation module respectively, and is used to calculate the proportion of the maximum value in the gradient difference sequence in the sum of all gradient differences to obtain the maximum data gradient ratio, and judge whether the data gradient ratio is less than the maximum data gradient ratio. If so, the data transmission to the gradient variance calculation module is maintained; if not, the data transmission to the gradient variance calculation module is interrupted.

8. The adaptive detection system for abnormal data of the Internet of Things as claimed in claim 5, characterized in that: When the abnormal data detection module does not include the data grouping unit and the first abnormal data extraction unit, the abnormal data detection module further includes: A graph drawing unit, used for drawing a two-dimensional line graph of the monitoring data value sequence according to the time flow, and drawing a straight line where the abnormal demarcation point threshold is located; The second abnormal data extraction unit is connected to the diagram drawing unit and is used to compare the number of data points on both sides of the straight line and extract the data points on the side with fewer data points as abnormal data.

Citation Information

Patent Citations

  • Method for monitoring safe operation of agricultural product conveying belt

    CN115684174A

  • Federal learning-based power data anomaly detection method

    CN116451117A