Abnormal data detection method and system

Abstract

The invention relates to the technical field of data processing, and discloses an abnormal data detection method and system.The method comprises the steps that a dynamic threshold range is obtained, an initial index value, a health index and a comprehensive score of information equipment are calculated, whether the comprehensive score is within the dynamic threshold range or not is judged, and if not, the upper limit and the lower limit of the dynamic threshold range are adjusted; obtaining an optimization threshold range; if yes, detecting the monitoring index real-time data according to the optimized threshold range or the dynamic threshold range to obtain a detection result; and according to comparison of the comprehensive score and the dynamic threshold interval, the upper limit and the lower limit of the dynamic threshold range are adjusted to obtain an optimized threshold range, and the interval size of the dynamic threshold range is expanded, so that the fluctuation data is covered in the optimized threshold range, the judgment error is reduced, and the detection accuracy of the abnormal data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to an abnormal data detection method and system. Background Art

[0002] In the modern information society, the stable operation of information equipment is crucial to business operations, personal life and even national security. Information equipment, including but not limited to servers, network equipment, storage devices, etc., their health status directly affects the overall performance and reliability of information systems. With the rapid development of information technology, the number and types of information equipment have increased dramatically, and with it the complexity of monitoring and managing information equipment. The traditional static threshold monitoring method, that is, setting a fixed threshold to evaluate the monitoring indicator data of information equipment, can no longer meet the monitoring needs of modern information systems. This is because the static threshold lacks flexibility and intelligence, and cannot adapt to the working status and environmental conditions of information equipment that change over time.

[0003] Specifically, the monitoring indicator data of information equipment usually shows regular fluctuations related to time. For example, the number of visits to the office application server shows a specific pattern during the working hours of the working day, while the traditional static threshold method is only set based on the system capacity, ignoring this time correlation, resulting in false positives or negative positives in certain periods. Especially in abnormal data detection, it is difficult to accurately distinguish normal fluctuations from true abnormalities by relying solely on fixed thresholds, which reduces the accuracy and efficiency of the monitoring system. In the prior art, by analyzing and predicting historical data, a prediction result is obtained, and a dynamic threshold interval is determined according to the maximum and minimum values ​​of the prediction result. This is a common practice, but the data is random and uncertain, and there are random fluctuations and noise. When the actual data is not in the dynamic threshold interval, it is judged as abnormal data. It is possible that the above data is affected by fluctuations rather than abnormal data. The dynamic threshold range determined by predicting the maximum and minimum values ​​cannot cover the fluctuating data, resulting in judgment errors in the detection process, resulting in the dynamic threshold range in the prior art being inaccurate. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides an abnormal data detection method and system to achieve dynamic threshold range optimization and abnormal data detection.

[0005] The present invention provides an abnormal data detection method, comprising: Step S1, obtaining monitoring index data of information equipment, and determining a dynamic threshold range for evaluating real-time monitoring index data according to the monitoring index data; Several information devices are connected to form an information monitoring system; Step S2, calculating the initial index value of the information equipment according to the monitoring index data; Step S3, calculating the health index according to the initial index value; Step S4, calculating a comprehensive score based on the initial indicator value and the health index; Step S5, determining whether the comprehensive score is within the dynamic threshold range, if not, executing step S6; if yes, executing step S7; Step S6, adjusting the upper limit or lower limit of the dynamic threshold range to obtain an optimized threshold range; Step S7, detecting the real-time data of the monitoring indicator according to the optimized threshold range or the dynamic threshold range to obtain a detection result.

[0006] Further, the initial indicator value of the information equipment is calculated according to the monitoring indicator data, specifically including: Step S21, filtering key indicator data from monitoring indicator data of information equipment; Step S22, determining the key indicator threshold of each information device according to the monitoring indicator data; Step S23, calculating the deviation between the key indicator data and the key indicator threshold; Step S24, determining the weight of the key indicator data according to the deviation; Step S25, calculating the initial index value of the information device according to the deviation and the weight.

[0007] Furthermore, based on the initial indicator value, the health index of the information device is calculated, specifically including: Step S31, taking the information device with the largest deviation between the key indicator data and the key indicator threshold as the main abnormal information device; Step S32, calculating the correlation coefficient between the deviation of other information devices in the information monitoring system and the deviation of the main abnormal information device; Step S33, defining connection priorities for other information devices according to the correlation coefficients; sorting the other information devices from high to low according to the correlation coefficients, and sequentially connecting the other information devices to the main abnormal information device according to the sorting results; Step S34, according to the connection priority, the main abnormal information device is connected to other information devices in sequence to form an abnormal propagation path; Step S35, determining a health index according to the initial indicator value under each abnormal propagation path; In the abnormal propagation path, the health index of other information devices with higher connection priority is the highest, and the health index of the main abnormal information is the lowest.

[0008] Further, step S35, determining the health index according to the abnormal propagation path, specifically includes: Step S351, calculating the length of the abnormal propagation path, and obtaining the shortest abnormal propagation path among the abnormal propagation paths; Step S352, determining a redundant path according to the abnormal propagation path and the shortest abnormal propagation path; Step S353, determining the connection type and the number of connections between different information devices according to the abnormal propagation path, the shortest abnormal propagation path, and the redundant path; Step S354, determining the complexity of the abnormal propagation path, the shortest abnormal propagation path, and the redundant path according to the connection type and the number of connections between different information devices; Step S355, calculating the degree of influence of each abnormal propagation path on the abnormal event according to the correlation coefficient and complexity; Step S356, determining the health index of each abnormal propagation path according to the impact degree and the initial indicator value.

[0009] Furthermore, the calculation formula of the comprehensive score is: ; In the formula, represents the health index of the ith abnormal propagation path, represents the initial index value of the ith abnormal propagation path, n represents the maximum number of abnormal propagation paths, represents the influence of the ith abnormal propagation path, Represents the adjustment factor.

[0010] Furthermore, the calculation formula for the degree of influence is: ; ; In the formula, represents the influence of the jth information device on the i-th abnormal propagation path, represents the correlation coefficient between the jth information device and other information devices in the i-th abnormal propagation path in terms of deviation, m represents the maximum number of information devices in the i-th abnormal propagation path, represents the complexity of the ith abnormal propagation path, represents the complexity of the shortest anomaly propagation path among the i-th anomaly propagation paths, represents the complexity of the redundant path in the i-th abnormal propagation path; T represents the connection type set, represents the weight of the jth information device belonging to the tth connection type in the connection type set, Represents the number of the jth information device in the i-th abnormal propagation path that belongs to the tth connection type in the connection type set.

[0011] Furthermore, when the comprehensive score is greater than the dynamic threshold range, the upper limit of the dynamic threshold range is increased; when the comprehensive score is less than the dynamic threshold range, the lower limit of the dynamic threshold range is decreased.

[0012] Furthermore, the real-time data is detected according to the optimized threshold range or the dynamic threshold range to obtain the detection result, which specifically includes: Step S71, obtaining real-time data of information equipment; Step S72, comparing the real-time data with the optimized threshold range or the dynamic threshold range; Step S73, if the real-time data is greater than or equal to the upper limit of the optimization threshold range or the dynamic threshold range, and less than or equal to the lower limit of the optimization threshold range or the dynamic threshold range, the real-time data is defined as normal data; if the real-time data is less than the upper limit of the optimization threshold range or the dynamic threshold range, or greater than the lower limit of the optimization threshold range or the dynamic threshold range, the real-time data is defined as fluctuating data; Step S74, calculate the abnormal score according to the fluctuation data; and set the abnormal score threshold. When the calculated abnormal score is greater than the abnormal score threshold, the fluctuation data is abnormal data; when the calculated abnormal score is less than or equal to the abnormal score threshold, the fluctuation data is normal data.

[0013] Furthermore, the abnormal score is calculated through historical data of the information device.

[0014] The present invention also provides an abnormal data detection system, which is used to execute the abnormal data detection method according to any one of claims 1 to 9, and comprises the following modules: Dynamic threshold range determination module: used to obtain monitoring indicator data of information equipment, and define a dynamic threshold range for evaluating the monitoring indicator data according to the monitoring indicator data of the information equipment; Initial indicator value calculation module: connected to the dynamic threshold range determination module, used to calculate the initial indicator value of the information device according to the monitoring indicator data; Health index calculation module: connected to the initial index value calculation module, used to calculate the health index of the information device according to the initial index value; Comprehensive score calculation module: connected with the initial index value calculation module and the health index calculation module, used to calculate the comprehensive score of the information device according to the initial index value and the health index; Judgment execution module: connected to the comprehensive score calculation module, used to judge whether the comprehensive score is within the dynamic threshold range, if not, execute step S6; if yes, execute step S7; Range optimization module: connected to the judgment execution module, used to adjust the upper and lower limits of the dynamic threshold range to obtain the optimized threshold range; Data detection module: connected with the range optimization module, detects the real-time data of the monitoring indicators according to the optimized threshold range or dynamic threshold range to obtain the detection results.

[0015] The embodiments of the present invention have the following technical effects: The present invention obtains a comprehensive score by comprehensively considering the initial indicator value and the health index. According to the comparison between the comprehensive score and the dynamic threshold interval, the upper and lower limits of the dynamic threshold range are adjusted to obtain the optimized threshold range, and the interval size of the dynamic threshold range is expanded so that the fluctuating data is also covered in the optimized threshold range, thereby reducing the judgment error and improving the detection accuracy of abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention; Figure 2 It is a structural diagram of the abnormal data detection method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the scope of protection of the present invention.

[0019] Figure 1 is a flow chart of an abnormal data detection method provided by an embodiment of the present invention. Figure 1 , specifically including: Step S1, obtaining monitoring index data of information equipment, and determining a dynamic threshold range for evaluating real-time monitoring index data according to the monitoring index data; First, several information devices are connected to form an information monitoring system to monitor the data of information devices. Secondly, information devices are: hardware devices used for information processing, storage, and transmission, including but not limited to servers, network devices (such as routers, switches), storage devices (disk arrays, SSDs), terminal devices (personal computers, workstations), etc. Monitoring indicator data include but are not limited to bandwidth utilization, packet loss rate, latency, CPU utilization, memory usage, disk read and write speed, request response time, transaction processing rate, error rate, query response time, number of login failures, number of security events, etc. Finally, monitoring indicator data is time series data, and regular fluctuations related to time will occur.

[0020] The historical monitoring indicator data in the information monitoring system is historical time series data.

[0021] In this embodiment, three months of historical time series data (time series data) are used to train the machine learning model, and the historical monitoring indicator data generated every hour is used as a data point, with a total of 2160 data, to output the future monitoring indicator data of the information equipment; the real-time data points are compared with the future monitoring indicator data, and the dynamic threshold range is determined according to the predicted monitoring indicator data, and the monitoring indicator data is evaluated according to the dynamic threshold range.

[0022] Before training, historical time series data needs to be preprocessed to remove outliers and missing values ​​to ensure the quality of training data.

[0023] Step S2, calculating the initial indicator value of the information device according to the monitoring indicator data, specifically includes: Step S21, filtering key indicator data from monitoring indicator data of information equipment; The key indicator data of information equipment is mainly screened based on the historical time series data and equipment functions. In this embodiment, the following monitoring indicator data are preferably used as the key indicator data: Key server indicators: CPU usage: an important indicator used to measure the processing capacity of the server. Too high CPU usage may indicate that the server is overloaded, affecting the response speed and stability. Memory usage: Memory is a key resource for servers to run applications and cache data. High memory usage may lead to performance degradation and response delays. Disk waiting time: The speed of disk I / O operations directly affects data access efficiency. Long I / O waiting times may mean disk bottlenecks, affecting server performance.

[0024] Key indicator data of network equipment: Bandwidth utilization: It is an important indicator of network equipment performance. High utilization may cause network congestion and delays, affecting user experience.

[0025] Packet error rate: reflects the reliability of network data transmission. A high error rate may indicate network quality problems or device failures. Device temperature: Temperature monitoring helps prevent overheating that could cause device failure or performance degradation.

[0026] Key indicators of storage devices: Read and write latency: reflects the response speed of the storage device. High latency may affect data access efficiency.

[0027] Device health status: Monitor the health of storage devices through SMART data to detect potential failures in a timely manner.

[0028] Storage utilization: Monitor storage space usage to avoid service interruptions due to insufficient storage space.

[0029] Key indicator data of terminal equipment: CPU temperature: Excessively high CPU temperature may affect device performance and hardware life.

[0030] Network connection quality: A stable network connection is a prerequisite for the normal operation of terminal devices. Poor connection quality may affect remote collaboration and data synchronization.

[0031] Step S22, determining the key indicator threshold of each information device according to the monitoring indicator data; The key indicator thresholds of information equipment are obtained by analyzing the historical monitoring indicator data. The key indicator thresholds cover the key indicator data range for the stable operation of information equipment.

[0032] Step S23, calculating the deviation between the key indicator data and the key indicator threshold; For each key indicator data, calculate the deviation between its actual data and the key indicator data range, and use the historical deviation, historical deviation frequency, and historical average deviation to calculate the deviation.

[0033] Step S24, determining the weight of the key indicator data according to the deviation; This embodiment assigns weights to each key indicator data based on the size of the deviation. The larger the deviation, the more important the indicator is, and the larger the weight should be. Further, this embodiment uses the square of the deviation as the weight.

[0034] Step S25, calculating the initial index value of the information device according to the deviation and the weight.

[0035] The initial index value is calculated for all key index data, and finally the initial index value of each information device is obtained by weighted summation. Furthermore, the initial index value of the abnormal propagation path of the information monitoring system is also calculated by weighted summation of the initial index value of each information device on the abnormal propagation path.

[0036] The calculation formula of the initial index value is: ; ; In the formula, represents the initial index value of the ith abnormal propagation path, Represents the weight of the key indicator data of the information equipment in the i-th abnormal propagation path, represents the deviation of the key indicator data of the information device in the ith abnormal propagation path, n represents the maximum number of abnormal propagation paths, m represents the maximum number of information devices in the ith abnormal propagation path, Represents the weight of the key indicator data of the jth information device in the i-th abnormal propagation path, Represents the deviation of the key indicator data of the jth information device in the i-th abnormal propagation path.

[0037] Step S3, calculating the health index according to the initial index value, specifically includes: Step S31, taking the information device with the largest deviation between the key indicator data and the key indicator threshold as the main abnormal information device; Step S32, calculating the correlation coefficient between the deviation of other information devices in the information monitoring system and the deviation of the main abnormal information device; The correlation coefficient is calculated using the Pearson correlation coefficient formula. The closer the correlation coefficient is to 1 and -1, the stronger the correlation between the two information devices is.

[0038] In addition, this embodiment also calculates the correlation coefficient of any two information devices among other information devices through the Pearson correlation coefficient formula.

[0039] Step S33, defining connection priorities for other information devices according to the correlation coefficients; sorting the other information devices from high to low according to the correlation coefficients, and sequentially connecting the other information devices to the main abnormal information device according to the sorting results; Step S34, according to the connection priority, the main abnormal information device is connected to other information devices in sequence to form an abnormal propagation path; The abnormal propagation path refers to the path through which the abnormal impact is transmitted to other information devices through the network or system architecture when an abnormality occurs in an information device. For example, a server fails, causing the performance of the network devices connected to it to degrade, which in turn affects other servers and terminal devices.

[0040] According to the abnormal propagation path formed by the connection between the main abnormal information device and other information devices, the corresponding abnormal propagation path is found in the topological structure of the information monitoring system.

[0041] Step S35, determining a health index according to the initial index value under each abnormal propagation path; In the abnormal propagation path, the health index of other information devices with higher connection priority is the highest, and the health index of the main abnormal information is the lowest.

[0042] The determination of health index includes: Step S351, calculating the length of the abnormal propagation path, and obtaining the shortest abnormal propagation path among the abnormal propagation paths; The graph theory algorithm is used to calculate the path length of the abnormal event from the main abnormal information device to the last other information device (target information device). The path length is determined by the number of information devices involved in the abnormal propagation path. From several abnormal propagation paths, the path involving the least number of information devices is selected as the shortest abnormal propagation path.

[0043] The shortest anomaly propagation path is found in the topology.

[0044] Step S352, determining a redundant path according to the abnormal propagation path and the shortest abnormal propagation path; Redundant paths refer to the existence of multiple paths among several abnormal propagation paths that can reach the same target information device, and the multiple paths include primary paths and redundant paths; The above multiple paths are analyzed based on path length, bandwidth, delay, load, and reliability, and the path with the best performance is used as the main path, and the other paths are redundant paths. When the main path fails, the redundant path can effectively carry the information flow and reduce the impact of abnormal events.

[0045] Redundant paths are found in the topology.

[0046] Step S353, determining the connection type and the number of connections between different information devices according to the abnormal propagation path, the shortest abnormal propagation path, and the redundant path; During the data detection process, the type and number of connections between information devices are crucial to assessing system complexity and potential abnormal propagation paths.

[0047] According to the information equipment, the connection types can be divided into: direct connection, multi-hop, loop, and star.

[0048] Counting the number of information device pairs in each connection type path helps evaluate the complexity of information devices. Specifically, for direct connection type paths, the number of active connections of information device ports is counted to obtain the number of connections; for multi-hop connection type paths, the number of connections transferred through a certain information device is counted as the number of connections; for loop connection type paths, first check the information device configuration, especially the interface configuration, to find directly connected information devices, then analyze the routing table and forwarding rules, determine the transmission rules between different information devices, identify indirectly connected information devices based on the transmission rules, and by comparing directly and indirectly connected information devices, find the information device group that forms a closed loop, and mark the information device group in the topology structure, so as to obtain the corresponding number of connections. For information devices on the star connection type path, all are directly connected to the target information device, and the number of connections is determined by the number of ports of the target information device.

[0049] Step S354, determining the complexity of the abnormal propagation path, the shortest abnormal propagation path, and the redundant path according to the connection type and the number of connections between different information devices; The complexity of the abnormal propagation path is determined according to its connection type. The complexity of the abnormal propagation path with a star connection type is greater than the complexity of the abnormal propagation path with a ring connection type. The complexity of the abnormal propagation path with a ring connection type is greater than the complexity of the abnormal propagation path with a multi-hop connection type. The complexity of the abnormal propagation path with a multi-hop connection type is greater than the complexity of the abnormal propagation path with a direct connection type.

[0050] The complexity of the shortest abnormal propagation path is determined by the number of information devices and the type of information device connection. When the connection type of information devices in the shortest abnormal propagation path is multi-hop, loop, or star, the number of corresponding information devices will also increase, and the complexity of this shortest abnormal propagation path is relatively high. If the connection type of information devices in the shortest abnormal propagation path is direct connection, the number of information devices involved is relatively small, and the complexity is relatively low.

[0051] The main path must pass through several information devices in the topology structure, which may increase the complexity of device management and maintenance; when the main path fails, the redundant path can effectively carry the information flow and reduce the impact of abnormal events. The redundant path has lower complexity.

[0052] Since the abnormal propagation path and the shortest propagation path have different complexities due to the different connection types, this embodiment assigns connection type weights according to the connection types of the abnormal propagation path and the shortest propagation path. The connection type weights of the abnormal propagation path and the shortest propagation path with a star connection type are the largest, and the connection type weights of the abnormal propagation path and the shortest propagation path with a ring connection type are second. The connection type weights of the abnormal propagation path and the shortest propagation path with a multi-hop connection type are smaller than the connection type weights of the abnormal propagation path and the shortest propagation path with a ring connection type, and the connection type weights of the abnormal propagation path and the shortest propagation path with a multi-hop connection type are larger than the connection type weights of the abnormal propagation path and the shortest propagation path with a direct connection type.

[0053] In this embodiment, the connection type weight of the redundant path is defined as the minimum, preferably 0.1.

[0054] In this embodiment, the connection type weight of the main path is defined between the connection type weights of the abnormal propagation path with multi-hop connection type and the shortest propagation path and the connection type weights of the abnormal propagation path with ring connection type and the shortest propagation path.

[0055] Step S355, calculating the degree of influence of each abnormal propagation path on the abnormal event according to the correlation coefficient and complexity; The calculation formula for the impact degree is: ; ; In the formula, represents the influence of the jth information device on the i-th abnormal propagation path, represents the correlation coefficient of the jth information device in the i-th abnormal propagation path with other information devices in terms of deviation. In this formula, the jth information device is defined as the main abnormal information device, m represents the maximum number of information devices in the i-th abnormal propagation path, represents the complexity of the ith abnormal propagation path, represents the complexity of the shortest anomaly propagation path among the i-th anomaly propagation paths, represents the complexity of the redundant path in the i-th abnormal propagation path; T represents the connection type set, represents the weight of the jth information device belonging to the tth connection type in the connection type set, Represents the number of the jth information device in the i-th abnormal propagation path that belongs to the tth connection type in the connection type set.

[0056] Further, It measures the additional complexity of the abnormal propagation path compared to the shortest propagation path. Close to , which means that the abnormal propagation path is the shortest propagation path; when If it is close to 1, it means that the information equipment in the current path is greatly affected by abnormal events.

[0057] It reflects the correction of redundant paths to the complexity of abnormal propagation paths. If there are a large number of redundant paths, that is, The impact will be relatively small, making the system more fault-tolerant.

[0058] The influence of each type of connection in the anomaly propagation path is considered. The more times different types of connections are repeated, the greater their contribution to the complexity of anomaly event propagation, increasing the degree of impact.

[0059] Step S356, determining the health index of each abnormal propagation path according to the impact degree and the initial indicator value.

[0060] The calculation formula of health index is: ; ; In the formula, represents the health index of the ith abnormal propagation path, represents the initial index value of the ith abnormal propagation path, represents the influence of the ith abnormal propagation path, represents the adjustment factor, Manual adjustment is required.

[0061] Step S4, calculating the comprehensive score of the information device according to the initial indicator value and the health index; The comprehensive scores of all information devices on each abnormal propagation path are weighted and summed to obtain the comprehensive score of each abnormal propagation path: ; ; In the formula, represents the comprehensive score of the ith abnormal propagation path, Represents the comprehensive score of the jth information device in the i-th abnormal propagation path.

[0062] Step S5, determining whether the comprehensive score is within the dynamic threshold range, if not, executing step S6; if yes, executing step S7; When the comprehensive score is greater than the upper limit of the dynamic threshold range, the upper limit of the dynamic threshold range is increased; when the comprehensive score is less than the dynamic threshold range, the lower limit of the dynamic threshold range is decreased.

[0063] In order to ensure that the numerical type or value of the comprehensive score is consistent with the dynamic threshold range, standardization is very necessary. Standardization can convert data of different dimensions and ranges to a unified standard scale. Standardization can be: minimum and maximum normalization, Z-Sore normalization, decimal calibration normalization, etc.

[0064] Step S6, adjusting the upper limit and the lower limit of the dynamic threshold range to obtain an optimized threshold range; In this embodiment, the dynamic threshold range is adjusted according to the comprehensive score, and the system can automatically adapt to changes in the system environment or data characteristics. For example, when a sudden increase in network traffic causes a general increase in future monitoring indicator data, the automatic adjustment of the upper limit of the dynamic threshold can avoid false alarms, and vice versa. This adaptability makes the system more robust and effective in the face of changing working conditions. Optimizing the threshold range can capture abnormal events more accurately.

[0065] Step S7, detecting the real-time data of the monitoring indicator according to the optimized threshold range or the dynamic threshold range to obtain the detection result, specifically including: Step S71, obtaining real-time data of information equipment; Step S72, comparing the real-time data with the optimized threshold range or the dynamic threshold range; Step S73, if the real-time data is greater than or equal to the upper limit of the optimization threshold range or the dynamic threshold range, and less than or equal to the lower limit of the optimization threshold range or the dynamic threshold range, the real-time data is defined as normal data; if the real-time data is less than the upper limit of the optimization threshold range or the dynamic threshold range, or greater than the lower limit of the optimization threshold range or the dynamic threshold range, the real-time data is defined as fluctuating data; Step S74, calculate the abnormal score according to the fluctuation data; and set the abnormal score threshold. When the calculated abnormal score is greater than the abnormal score threshold, the fluctuation data is abnormal data; when the calculated abnormal score is less than or equal to the abnormal score threshold, the fluctuation data is normal data.

[0066] The abnormal score is calculated by using the historical monitoring index data of the information device. The abnormal score threshold is obtained by analyzing the historical monitoring index data of the information device.

[0067] The calculation process of the anomaly score is as follows: 1. Deviation A In this embodiment, the degree to which real-time data deviates from the upper or lower limit of the optimization threshold range or the dynamic threshold range is defined as deviation A. Deviation A is a value greater than 0 and less than 1. The larger the value, the greater the deviation of the actual indicator from the optimization threshold range or the dynamic threshold range.

[0068] ; in, Represents the difference between the real-time data and the upper or lower limit of the optimized threshold range or dynamic threshold range. Represents the difference between the future monitoring indicator data (predicted value) and the upper or lower limit of the optimized threshold range or dynamic threshold range.

[0069] 2. Historical deviation frequency B Considering that there are many factors affecting the monitoring indicators of information equipment, historical monitoring indicator data has a certain degree of randomness. Usually, the future monitoring indicator data obtained by the machine learning model cannot all be within the optimization threshold range or the dynamic threshold range. The deviation frequency of the historical monitoring indicator data from the upper or lower limit of the optimization threshold range or the dynamic threshold range is calculated through the historical deviation frequency B: ; In the formula, represents the historical deviation frequency, W represents the number of historical monitoring indicator data, and F represents the number of real-time data that deviates from the optimization threshold range or dynamic threshold range; B is a floating point number greater than 0 and less than 1. The larger the B, the smaller the probability of deviation. If deviation occurs, the degree of abnormality should be high. The smaller the B, the weaker the regularity. If deviation occurs, the degree of abnormality should be appropriately reduced.

[0070] 3. Historical average deviation C In addition to the frequency of historical deviations, we should also pay attention to the magnitude of each deviation, and introduce the historical average deviation C. Calculate the deviation A for each historical data deviation point, and you will get A 1 , A 2 , A 3 …A F , calculate the historical average deviation C: ; 4. Abnormal score Z The abnormal score Z of the current real-time data is: .

[0071] By integrating the analysis of real-time data and historical data, more accurate anomaly detection capabilities are provided, while also enhancing the system's adaptability and robustness.

[0072] Figure 2 is a structural diagram of an abnormal data detection method provided by an embodiment of the present invention. Figure 2 , including the following modules: Dynamic threshold range determination module: used to obtain monitoring indicator data of information equipment, and define a dynamic threshold range for evaluating the monitoring indicator data according to the monitoring indicator data of the information equipment; Initial indicator value calculation module: connected to the dynamic threshold range determination module, used to calculate the initial indicator value of the information device according to the monitoring indicator data; Health index calculation module: connected to the initial index value calculation module, used to calculate the health index of the information device according to the initial index value; Comprehensive score calculation module: connected with the initial index value calculation module and the health index calculation module, used to calculate the comprehensive score of the information device according to the initial index value and the health index; Judgment execution module: connected to the comprehensive score calculation module, used to judge whether the comprehensive score is within the dynamic threshold range. If not, the upper and lower limits of the dynamic threshold range are adjusted to obtain the optimized threshold range; if so, the real-time data of the monitoring indicators is tested according to the optimized threshold range or the dynamic threshold range to obtain the test results; Range optimization module: connected to the judgment execution module, used to adjust the upper and lower limits of the dynamic threshold range to obtain the optimized threshold range; Data detection module: connected with the range optimization module, detects the real-time data of the monitoring indicators according to the optimized threshold range or dynamic threshold range to obtain the detection results.

[0073] This embodiment also provides an electronic device, including one or more processors and a memory.

[0074] The processor may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 500 to perform desired functions.

[0075] The memory may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may run the program instructions to implement the abnormal data detection method of any embodiment of the present application described above and / or other desired functions. Various contents such as initial external parameters, thresholds, etc. may also be stored in the computer-readable storage medium.

[0076] In one example, the electronic device may further include: an input device and an output device, and these components are interconnected through a bus system and / or other forms of connection mechanisms (not shown). The input device may include, for example, a keyboard, a mouse, etc. The output device may output various information to the outside, including early warning information, braking force, etc. The output device may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0077] In addition, the electronic device may further include any other appropriate components according to specific application scenarios.

[0078] In addition to the above-mentioned methods and devices, an embodiment of the present application may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the abnormal data detection method provided by any embodiment of the present application.

[0079] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages, such as Java, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0080] In addition, an embodiment of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes the steps of the abnormal data detection method provided by any embodiment of the present application.

[0081] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0082] It should be noted that the terms used in the present invention are only for describing specific embodiments, rather than limiting the scope of the present application. As shown in the present specification, unless the context clearly indicates an exception, the words "one", "a", "a kind of" and / or "the" do not specifically refer to the singular, but may also include the plural. The terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method or device. In the absence of more restrictions, the elements defined by the sentence "include one..." do not exclude the presence of other identical elements in the process, method or device including the elements.

[0083] It should also be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", etc. should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be a connection between the two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. Abnormal data detection methods, It is characterized in that include: Step S1, obtaining monitoring index data of information equipment, and determining a dynamic threshold range for evaluating real-time monitoring index data according to the monitoring index data; Several information devices are connected to form an information monitoring system; Step S2, calculating the initial index value of the information equipment according to the monitoring index data; Step S3, calculating the health index according to the initial index value; Step S4, calculating a comprehensive score based on the initial indicator value and the health index; Step S5, determining whether the comprehensive score is within the dynamic threshold range, if not, executing step S6; if yes, executing step S7; Step S6, adjusting the upper limit or lower limit of the dynamic threshold range to obtain an optimized threshold range; Step S7, detecting the real-time data of the monitoring index according to the optimized threshold range or the dynamic threshold range to obtain the detection result.

2. The abnormal data detection method according to claim 1, It is characterized in that Calculate the initial indicator value of the information equipment based on the monitoring indicator data, including: Step S21, filtering key indicator data from monitoring indicator data of information equipment; Step S22, determining the key indicator threshold of each information device according to the monitoring indicator data; Step S23, calculating the deviation between the key indicator data and the key indicator threshold; Step S24, determining the weight of the key indicator data according to the deviation; Step S25, calculating the initial index value of the information device according to the deviation and the weight.

3. The abnormal data detection method according to claim 1, It is characterized in that According to the initial index value, the health index of the information equipment is calculated, including: Step S31, taking the information device with the largest deviation between the key indicator data and the key indicator threshold as the main abnormal information device; Step S32, calculating the correlation coefficient between the deviation of other information devices in the information monitoring system and the deviation of the main abnormal information device; Step S33, defining connection priorities for other information devices according to the correlation coefficients; sorting the other information devices from high to low according to the correlation coefficients, and sequentially connecting the other information devices to the main abnormal information device according to the sorting results; Step S34, according to the connection priority, the main abnormal information device is connected to other information devices in sequence to form an abnormal propagation path; Step S35, determining a health index according to the initial index value under each abnormal propagation path; In the abnormal propagation path, the health index of other information devices with the highest connection priority is the highest, and the health index of the main abnormal information device is the lowest.

4. The abnormal data detection method according to claim 3, It is characterized in that Step S35, determining the health index according to the initial indicator value under the abnormal propagation path, specifically includes: Step S351, calculating the length of the abnormal propagation path, and obtaining the shortest abnormal propagation path among the abnormal propagation paths; Step S352, determining a redundant path according to the abnormal propagation path and the shortest abnormal propagation path; Step S353, determining the connection type and the number of connections between different information devices according to the abnormal propagation path, the shortest abnormal propagation path, and the redundant path; Step S354, determining the complexity of the abnormal propagation path, the shortest abnormal propagation path, and the redundant path according to the connection type and the number of connections between different information devices; Step S355, calculating the degree of influence of each abnormal propagation path on the abnormal event according to the correlation coefficient and complexity; Step S356, determining the health index of each abnormal propagation path according to the impact degree and the initial indicator value.

5. The abnormal data detection method according to claim 4, It is characterized in that The calculation formula of the health index is: ； In the formula, represents the health index of the ith abnormal propagation path, represents the initial index value of the ith abnormal propagation path, n represents the maximum number of abnormal propagation paths, represents the influence of the ith abnormal propagation path, Represents the adjustment factor.

6. The abnormal data detection method according to claim 5, It is characterized in that The calculation formula for the impact degree is: ； ； In the formula, represents the influence of the jth information device on the i-th abnormal propagation path, represents the correlation coefficient between the jth information device and other information devices in the i-th abnormal propagation path in terms of deviation, m represents the maximum number of information devices in the i-th abnormal propagation path, represents the complexity of the ith abnormal propagation path, represents the complexity of the shortest anomaly propagation path among the i-th anomaly propagation paths, represents the complexity of the redundant path in the i-th abnormal propagation path; T represents the connection type set, represents the weight of the jth information device belonging to the tth connection type in the connection type set, Represents the number of the jth information device in the i-th abnormal propagation path that belongs to the tth connection type in the connection type set.

7. The abnormal data detection method according to claim 1, It is characterized in that When the comprehensive score is greater than the dynamic threshold range, the upper limit of the dynamic threshold range is increased; when the comprehensive score is less than the dynamic threshold range, the lower limit of the dynamic threshold range is decreased.

8. The abnormal data detection method according to claim 7, It is characterized in that The real-time data is tested according to the optimized threshold range or the dynamic threshold range to obtain the test results, including: Step S71, obtaining real-time data of information equipment; Step S72, comparing the real-time data with the optimized threshold range or the dynamic threshold range; Step S73, if the real-time data is greater than or equal to the upper limit of the optimization threshold range or the dynamic threshold range, and less than or equal to the lower limit of the optimization threshold range or the dynamic threshold range, the real-time data is defined as normal data; if the real-time data is less than the upper limit of the optimization threshold range or the dynamic threshold range, or greater than the lower limit of the optimization threshold range or the dynamic threshold range, the real-time data is defined as fluctuating data; Step S74, calculate the abnormal score according to the fluctuation data; and set the abnormal score threshold. When the calculated abnormal score is greater than the abnormal score threshold, the fluctuation data is abnormal data; when the calculated abnormal score is less than or equal to the abnormal score threshold, the fluctuation data is normal data.

9. The abnormal data detection method according to claim 8, It is characterized in that The abnormal score is calculated through historical data of the information device.

10. An abnormal data detection system, configured to execute the abnormal data detection method according to any one of claims 1 to 9, It is characterized in that Includes the following modules: Dynamic threshold range determination module: used to obtain monitoring indicator data of information equipment, and define a dynamic threshold range for evaluating the monitoring indicator data according to the monitoring indicator data of the information equipment; Initial indicator value calculation module: connected to the dynamic threshold range determination module, used to calculate the initial indicator value of the information device according to the monitoring indicator data; Health index calculation module: connected to the initial index value calculation module, used to calculate the health index of the information device according to the initial index value; Comprehensive score calculation module: connected with the initial index value calculation module and the health index calculation module, used to calculate the comprehensive score of the information device according to the initial index value and the health index; Judgment execution module: connected to the comprehensive score calculation module, used to judge whether the comprehensive score is within the dynamic threshold range. If not, the upper limit or lower limit of the dynamic threshold range is adjusted to obtain the optimized threshold range; If so, the real-time data of the monitoring indicator is tested according to the optimized threshold range or the dynamic threshold range to obtain the test result; Range optimization module: connected to the judgment execution module, used to adjust the upper and lower limits of the dynamic threshold range to obtain the optimized threshold range; Data detection module: connected with the range optimization module, detects the real-time data of the monitoring indicators according to the optimized threshold range or dynamic threshold range to obtain the detection results.

Images